Chapter_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Chapter 2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/rnarkk/PPBMH/blob/main/Chapter_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "phBEJ8iLIAwF"
      },
      "source": [
        "# Probabilistic Programming and Bayesian Methods for Hackers Chapter  2\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter2_MorePyMC/Ch2_MorePyMC_TFP.ipynb\"><img height=\"32px\" src=\"https://colab.research.google.com/img/colab_favicon.ico\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter2_MorePyMC/Ch2_MorePyMC_TFP.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>\n",
        "<br>\n",
        "<br>\n",
        "<br>\n",
        "\n",
        "Original content ([this Jupyter notebook](https://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter2_MorePyMC/Ch2_MorePyMC_PyMC2.ipynb)) created by Cam Davidson-Pilon ([`@Cmrn_DP`](https://twitter.com/Cmrn_DP))\n",
        "\n",
        "Welcome to Bayesian Methods for Hackers. The full Github repository is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). The other chapters can be found on the project's [homepage](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/). We hope you enjoy the book, and we encourage any contributions!\n",
        "___\n",
        "\n",
        "### Table of Contents\n",
        "\n",
        "- A little more on TFP\n",
        "  - TFP Variables\n",
        "    - Initializing Stochastic Variables\n",
        "    - Deterministic variables\n",
        "  - Combining with Tensorflow Core\n",
        "  - Including observations in the Model\n",
        "- Modeling approaches\n",
        "  - Same story; different ending\n",
        "  - Example: Bayesian A/B testing\n",
        "  - A Simple Case\n",
        "    - Execute the TF graph to sample from the posterior\n",
        "  - A and B together\n",
        "    - Execute the TF graph to sample from the posterior\n",
        "- An algorithm for human deceit\n",
        "  - The Binomial Distribution\n",
        "  - Example: Cheating among students\n",
        "    - Execute the TF graph to sample from the posterior\n",
        "  - Alternative TFP Model\n",
        "    - Execute the TF graph to sample from the posterior\n",
        "  - More TFP Tricks\n",
        "  - Example: Challenger Space Shuttle Disaster\n",
        "    - Normal Distributions\n",
        "      - Execute the TF graph to sample from the posterior\n",
        "    - What about the day of the Challenger disaster?\n",
        "    - Is our model appropriate?\n",
        "      - Execute the TF graph to sample from the posterior\n",
        "  - Exercises\n",
        "  - References\n",
        "___\n",
        "\n",
        "This chapter introduces more TFP syntax and variables and ways to think about how to model a system from a Bayesian perspective. It also contains tips and data visualisation techniques for assessing goodness-of-fit for your Bayesian model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5jrjUW3K-CK3",
        "outputId": "22d04ae3-16ae-4506-953e-99b5a4821d17"
      },
      "source": [
        "!pip3 install -Uq pip\n",
        "!pip3 uninstall -y tensorflow tensorflow-probability\n",
        "!pip3 install -Uq jax jaxlib oryx tfp-nightly[jax]\n",
        "\n",
        "!pip3 install -q wget"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[33mWARNING: Skipping tensorflow as it is not installed.\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[33mWARNING: Skipping tensorflow-probability as it is not installed.\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jFKYjxy1IAwG"
      },
      "source": [
        "\"\"\"\n",
        "The book uses a custom matplotlibrc file, which provides the unique styles for\n",
        "matplotlib plots. If executing this book, and you wish to use the book's\n",
        "styling, provided are two options:\n",
        "    1. Overwrite your own matplotlibrc file with the rc-file provided in the\n",
        "       book's styles/ dir. See http://matplotlib.org/users/customizing.html\n",
        "    2. Also in the styles is  bmh_matplotlibrc.json file. This can be used to\n",
        "       update the styles in only this notebook. Try running the following code:\n",
        "\n",
        "        import json\n",
        "        s = json.load(open(\"../styles/bmh_matplotlibrc.json\"))\n",
        "        matplotlib.rcParams.update(s)\n",
        "\"\"\"\n",
        "from jax import jit, random\n",
        "from jax.numpy import *\n",
        "from jax.numpy import int32 as i32, float32 as f32\n",
        "from jax.random import exponential, poisson, uniform\n",
        "from oryx import bijectors as tfb\n",
        "from oryx.distributions import (\n",
        "    Bernoulli, Binomial, Deterministic, Exponential, Gamma, Normal, Poisson, Uniform)\n",
        "from tensorflow_probability.substrates import jax as tfp\n",
        "\n",
        "from IPython.core.pylabtools import figsize\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "import wget\n",
        "\n",
        "plt.style.use('fivethirtyeight')\n",
        "\n",
        "%config InlineBackend.figure_format = 'retina'\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "sns.set_context('notebook')\n",
        "\n",
        "TFColor = [\n",
        "    '#F15854',\n",
        "    '#5DA5DA',\n",
        "    '#FAA43A',\n",
        "    '#60BD68',\n",
        "    '#F17CB0',\n",
        "    '#B2912F',\n",
        "    '#B276B2',\n",
        "    '#DECF3F',\n",
        "    '#4D4D4D']"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "24368dz9IAwM"
      },
      "source": [
        "## A little more on TensorFlow and TensorFlow Probability\n",
        "\n",
        "To explain TensorFlow Probability, it's worth going into the various methods of working with Tensorflow tensors. Here, we introduce the notion of Tensorflow graphs and how we can use certain coding patterns to make our tensor-processing workflows much faster and more elegant. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tOj8bjBmNuxm"
      },
      "source": [
        "### TensorFlow Graph and Eager Modes\n",
        "\n",
        "TFP accomplishes most of its heavy lifting via the main `tensorflow` library. The `tensorflow` library also contains many of the familiar computational elements of NumPy and uses similar notation. While NumPy directly executes computations (e.g. when you run `a + b`), `tensorflow` in graph mode instead builds up a \"compute graph\" that tracks that you want to perform the `+` operation on the elements `a` and `b`. Only when you evaluate a `tensorflow` expression  does the computation take place--`tensorflow` is lazy evaluated. The benefit of using Tensorflow over NumPy is that the graph enables mathematical optimizations (e.g. simplifications), gradient calculations via automatic differentiation, compiling the entire graph to C to run at machine speed, and also compiling it to run on a GPU or TPU. \n",
        "\n",
        "Fundamentally, TensorFlow uses [graphs](https://www.tensorflow.org/guide/graphs) for computation, wherein the graphs represent computation as dependencies among individual operations. In the programming paradigm for Tensorflow graphs, we first define the dataflow graph, and then create a TensorFlow session to run parts of the graph. A Tensorflow [`tf.Session()`](https://www.tensorflow.org/api_docs/python/tf/Session) object runs the graph to get the variables we want to model. In the example below, we are using a global session object `sess`, which we created above in the \"Imports and Global Variables\" section. \n",
        "\n",
        "To avoid the sometimes confusing aspects of lazy evaluation, Tensorflow's eager mode does immediate evaluation of results to give an even more similar feel to working with NumPy. With Tensorflow [eager](https://www.tensorflow.org/guide/eager) mode, you can evaluate operations immediately, without explicitly building graphs: operations return concrete values instead of constructing a computational graph to run later. If we're in eager mode, we are presented with tensors that can be converted to NumPy array equivalents immediately. Eager mode makes it easy to get started with TensorFlow and debug models.\n",
        "\n",
        "\n",
        "TFP is essentially:\n",
        "\n",
        "* a collection of tensorflow symbolic expressions for various probability distributions that are combined into one big compute graph, and\n",
        "* a collection of inference algorithms that use that graph to compute probabilities and gradients.\n",
        "\n",
        "For practical purposes, what this means is that in order to build certain models we sometimes have to use core Tensorflow. This simple example for Poisson sampling is how we might work with both graph and eager modes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CmiGas0kXiEw",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 240
        },
        "outputId": "75f36822-9bb6-43a2-a3eb-2c3220fdc9d1"
      },
      "source": [
        "key = random.PRNGKey(0)\n",
        "parameter = Exponential(rate = 1.)\n",
        "data_generator = poisson(key, rate = parameter)\n",
        "\n",
        "print(\"Value of sample from data generator random variable:\", data_generator)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-19-8e0cd33795de>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPRNGKey\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mparameter\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mexponential\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrate\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1.\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mdata_generator\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpoisson\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrate\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparameter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Value of sample from data generator random variable:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata_generator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mTypeError\u001b[0m: exponential() got an unexpected keyword argument 'rate'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9kArT4GTIAwT"
      },
      "source": [
        "In graph mode, Tensorflow will automatically assign any variables to a graph; they can then be evaluated in a session or made available in eager mode. If you try to define a variable when the session is already closed or in a finalised state, you will get an error. In the \"Imports and Global Variables\" section, we defined a particular type of session, called [`InteractiveSession`](https:///www.tensorflow.org/api_docs/python/tf/InteractiveSession). \n",
        "This definition of a global `InteractiveSession` allows us to access our session variables interactively via a shell or notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4IEk40NbIAwX"
      },
      "source": [
        "# !TODO Erase below\n",
        "Each of the tensors corresponds to a NumPy-like output. To distinguish the tensors from their NumPy-like counterparts, we will use the convention of appending an underscore to the version of the tensor that one can use NumPy-like arrays on. In other words, the output of `evaluate()` gets named as `variable` + `_` = `variable_` . Now, we can do our Poisson sampling using both the `evaluate()` function and this new convention for naming Python variables in TFP."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bk-vyPB9IAwX"
      },
      "source": [
        "# Assumptions\n",
        "parameter = Exponential(rate = 1.).sample(seed = random.PRNGKey(0))\n",
        "\n",
        "print(\"Sample from exponential distribution before evaluation:\", parameter)\n",
        "print(\"Evaluated sample from exponential distribution:\", parameter)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZlGWIiPLIAwo"
      },
      "source": [
        "More generally, we can use our `evaluate()` function to convert between the Tensorflow `tensor` data type and one that we can run operations on:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1tzQmnsFIAwp"
      },
      "source": [
        "print(\"'parameter' evaluated Tensor:\", parameter)\n",
        "print(\"'data_generator' sample evaluated Tensor:\", data_generator)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m0PLxpCIc--r"
      },
      "source": [
        "\n",
        "A general rule of thumb for programming in TensorFlow is that if you need to do any array-like calculations that would require NumPy functions, you should use their equivalents in TensorFlow. This practice is necessary because NumPy can produce only constant values but TensorFlow tensors are a dynamic part of the computation graph. If you mix and match these the wrong way, you will typically get an error about incompatible types."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wqkS8vztNoyh"
      },
      "source": [
        "### TFP Distributions\n",
        "\n",
        "Let's look into how [`tfp.distributions`](https://www.tensorflow.org/probability/api_docs/python/tfp/distributions) work.\n",
        "\n",
        "TFP uses distribution subclasses to represent *stochastic*, random variables. A variable is stochastic when the following is true: even if you knew all the values of the variable's parameters and components, it would still be random. Included in this category are instances of classes [`Poisson`](https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/Poisson), [`Uniform`](https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/Uniform), and [`Exponential`](https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/Exponential).\n",
        "\n",
        "You can draw random samples from a stochastic variable. When you draw samples, those samples become [`tf.Tensors`](https://www.tensorflow.org/api_docs/python/tf/Tensor) that behave deterministically from that point on. A quick mental check to determine if something is *deterministic* is: *If I knew all of the inputs for creating the variable `foo`, I could calculate the value of `foo`.*  You can add, subtract, and otherwise manipulate the tensors in a variety of ways discussed below. These operations are almost always deterministic.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NdKiqWtWIAwy"
      },
      "source": [
        "#### Initialise a Distribution\n",
        "\n",
        "Initialise a stochastic, or random variable requires a few class-specific parameters that describe the Distribution's shape, such as the location and scale. For example:\n",
        "\n",
        "```python\n",
        "some_distribution = Uniform(low = 0., high = 4.)\n",
        "```\n",
        "\n",
        "initialises a [`Uniform`](https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/Uniform) distribution with the lower bound at 0 and upper bound at 4. Calling `sample()` on the distribution returns a tensor that will behave deterministically from that point on:\n",
        "\n",
        "```python\n",
        "sampled_tensor = some_distribution.sample()\n",
        "```\n",
        "\n",
        "The next example demonstrates what we mean when we say that distributions are stochastic but tensors are deterministic:\n",
        "\n",
        "```\n",
        "derived_tensor_1 = 1 + sampled_tensor\n",
        "derived_tensor_2 = 1 + sampled_tensor  # equal to 1\n",
        "\n",
        "derived_tensor_3 = 1 + some_distribution.sample()\n",
        "derived_tensor_4 = 1 + some_distribution.sample()  # different from 3\n",
        "```\n",
        "\n",
        "The first two lines produce the same value because they refer to the same sampled tensor. The last two lines likely produce different values because they refer to independent samples drawn from the same distribution.\n",
        "\n",
        "To define a multiviariate distribution, just pass in arguments with the shape you want the output to be when creating the distribution. For example:\n",
        "\n",
        "```python\n",
        "betas = Uniform((0., 0.), (1., 1.))\n",
        "```\n",
        "\n",
        "Creates a Distribution with batch_shape (2,). Now, when you call betas.sample(),\n",
        "two values will be returned instead of one. You can read more about TFP shape semantics in the [TFP docs](https://github.com/tensorflow/probability/blob/master/tensorflow_probability/examples/jupyter_notebooks/Understanding_TensorFlow_Distributions_Shapes.ipynb), but most uses in this book should be self-explanatory."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UPt9k8YrIAwz"
      },
      "source": [
        "#### Deterministic variables\n",
        "\n",
        "We can create a deterministic distribution similarly to how we create a stochastic distribution. We simply call up the [`Deterministic`](https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/Deterministic) class from Tensorflow Distributions and pass in the deterministic value that we desire\n",
        "```python\n",
        "deterministic_variable = Deterministic(loc = some_function_of_variables)\n",
        "```\n",
        "\n",
        "Calling `tfd.Deterministic` is useful for creating distributions that always have the same value. However, the much more common pattern for working with deterministic variables in TFP is to create a tensor or sample from a distribution:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "feDM_HX6IAw0"
      },
      "source": [
        "# Stochastic variables\n",
        "λ_1 = Exponential(rate = 1.)\n",
        "λ_2 = Exponential(rate = 1.)\n",
        "τ = Uniform(low = 0., high = 10.)\n",
        "\n",
        "# Deterministic variable; getting results of lambda's after sampling    \n",
        "deterministic_variable = Deterministic(\n",
        "    loc = λ_1.sample(random.PRNGKey(0)) + λ_2.sample(random.PRNGKey(0)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cRzLJmAJIAw3"
      },
      "source": [
        "The use of the deterministic variable was seen in the previous chapter's text-message example.  Recall the model for $\\lambda$ looked like: \n",
        "\n",
        "$$\n",
        "\\lambda = \n",
        "\\begin{cases}\\lambda_1  & \\text{if } t \\lt \\tau \\cr\n",
        "\\lambda_2 & \\text{if } t \\ge \\tau\n",
        "\\end{cases}\n",
        "$$\n",
        "\n",
        "And in TFP code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IXdTQeqrIAw3"
      },
      "source": [
        "# Days\n",
        "num_data_points = 5  # in CH1 we had ~70 data points\n",
        "idx = arange(num_data_points)\n",
        "# for num_data_points samples, select from λ_2 if sampled τ >= day value, λ_1 otherwise\n",
        "rv_λ_deterministic = Deterministic(\n",
        "    array(\n",
        "        [λ_1.sample(random.PRNGKey(0)),\n",
        "         λ_2.sample(random.PRNGKey(0))])\n",
        "        [(τ.sample(random.PRNGKey(0)) >= idx).astype(i32)])\n",
        "λ_deterministic = rv_λ_deterministic.sample(random.PRNGKey(0))\n",
        "\n",
        "print(f\"{num_data_points} samples from our deterministic lambda model:\\n   \", λ_deterministic)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EFgJwLATIAw8"
      },
      "source": [
        "Clearly, if $\\tau, \\lambda_1$ and $\\lambda_2$ are known, then $\\lambda$ is known completely, hence it is a deterministic variable. We use indexing here to switch from $\\lambda_1$ to $\\lambda_2$ at the appropriate time. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IMNdtRTtIAxB"
      },
      "source": [
        "### Including observations in the model\n",
        "\n",
        "At this point, it may not look like it, but we have fully specified our priors. For example, we can ask and answer questions like \"What does my prior distribution of $\\lambda_1$ look like?\" \n",
        "\n",
        "To do this, we will sample from the distribution. The method `.sample()` has a very simple role: get data points from the given distribution. We can then evaluate the resulting tensor to get a NumPy array-like object. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VNdQVTSFIAxC"
      },
      "source": [
        "# Define our observed samples\n",
        "rv_λ_1 = Exponential(rate = 1.)\n",
        "λ_1 = rv_λ_1.sample(sample_shape = (20000,), seed = random.PRNGKey(0))\n",
        "    \n",
        "# Visualise stepwise prior distribution\n",
        "plt.figure(figsize(12.5, 5))\n",
        "plt.hist(λ_1, bins = 70, histtype = 'stepfilled')\n",
        "plt.title(r\"Prior distribution for $\\lambda_1$\")\n",
        "plt.xlim(0, 8);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9nOs9Gq3IAxH"
      },
      "source": [
        "To frame this in the notation of the first chapter, though this is a slight abuse of notation, we have specified $P(A)$. Our next goal is to include data/evidence/observations $X$ into our model. \n",
        "\n",
        "Sometimes we may want to match a property of our distribution to a property of observed data. To do so, we get the parameters for our distribution fom the data itself. In this example, the Poisson rate (average number of events) is explicitly set to one over the average of the data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qtHXSR6QIAxH"
      },
      "source": [
        "data = array([10., 5.], dtype = f32)\n",
        "rv_poisson = Poisson(rate = 1. / mean(data))\n",
        "poisson = rv_poisson.sample(random.PRNGKey(0))\n",
        "\n",
        "print(\"Two predetermined data points:\\n    \", data)\n",
        "print(\"Mean of data:\\n    \", mean(data))\n",
        "print(\"Random sample from poisson distribution with the mean as the poisson's rate:\\n    \", poisson)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8oxo5VcbIAxP"
      },
      "source": [
        "## Modeling approaches\n",
        "\n",
        "A good starting thought to Bayesian modeling is to think about *how your data might have been generated*. Position yourself in an omniscient position, and try to imagine how *you* would recreate the dataset. \n",
        "\n",
        "In the last chapter we investigated text message data. We begin by asking how our observations may have been generated:\n",
        "\n",
        "1.  We started by thinking \"what is the best random variable to describe this count data?\" A Poisson random variable is a good candidate because it can represent count data. So we model the number of sms's received as sampled from a Poisson distribution.\n",
        "\n",
        "2.  Next, we think, \"Ok, assuming sms's are Poisson-distributed, what do I need for the Poisson distribution?\" Well, the Poisson distribution has a parameter $\\lambda$. \n",
        "\n",
        "3.  Do we know $\\lambda$? No. In fact, we have a suspicion that there are *two* $\\lambda$ values, one for the earlier behaviour and one for the later behaviour. We don't know when the behaviour switches though, but call the switchpoint $\\tau$.\n",
        "\n",
        "4. What is a good distribution for the two $\\lambda$s? The exponential is good, as it assigns probabilities to positive real numbers. Well the exponential distribution has a parameter too, call it $\\alpha$.\n",
        "\n",
        "5.  Do we know what the parameter $\\alpha$ might be? No. At this point, we could continue and assign a distribution to $\\alpha$, but it's better to stop once we reach a set level of ignorance: whereas we have a prior belief about $\\lambda$, (\"it probably changes over time\", \"it's likely between 10 and 30\", etc.), we don't really have any strong beliefs about $\\alpha$. So it's best to stop here. \n",
        "\n",
        "    What is a good value for $\\alpha$ then? We think that the $\\lambda$s are between 10-30, so if we set $\\alpha$ really low (which corresponds to larger probability on high values) we are not reflecting our prior well. Similar, a too-high alpha misses our prior belief as well. A good idea for $\\alpha$ as to reflect our belief is to set the value so that the mean of $\\lambda$, given $\\alpha$, is equal to our observed mean. This was shown in the last chapter.\n",
        "\n",
        "6. We have no expert opinion of when $\\tau$ might have occurred. So we will suppose $\\tau$ is from a discrete uniform distribution over the entire timespan.\n",
        "\n",
        "\n",
        "Below we give a graphical visualisation of this, where arrows denote `parent-child` relationships. (provided by the [Daft Python library](http://daft-pgm.org/) )\n",
        "\n",
        "<img src=\"http://i.imgur.com/7J30oCG.png\">\n",
        "\n",
        "\n",
        "TFP and other probabilistic programming languages have been designed to tell these data-generation *stories*. More generally, B. Cronin writes [2]:\n",
        "\n",
        "> Probabilistic programming will unlock narrative explanations of data, one of the holy grails of business analytics and the unsung hero of scientific persuasion. People think in terms of stories - thus the unreasonable power of the anecdote to drive decision-making, well-founded or not. But existing analytics largely fails to provide this kind of story; instead, numbers seemingly appear out of thin air, with little of the causal context that humans prefer when weighing their options."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3RJEK_yjIAxR"
      },
      "source": [
        "### Same story; different ending.\n",
        "\n",
        "Interestingly, we can create *new datasets* by retelling the story.\n",
        "For example, if we reverse the above steps, we can simulate a possible realization of the dataset.\n",
        "\n",
        "1\\. Specify when the user's behaviour switches by sampling from $\\text{DiscreteUniform}(0, 80)$:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ma56S7r1IAxS"
      },
      "source": [
        "τ = uniform(random.PRNGKey(0), shape = (1,), minval = 0, maxval = 80)\n",
        "\n",
        "print(\"Value of τ (randomly taken from DiscreteUniform(0, 80)):\\n   \", τ)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xt_6sYG6IAxW"
      },
      "source": [
        "2\\. Draw $\\lambda_1$ and $\\lambda_2$ from a $\\text{Gamma}(\\alpha)$ distribution:\n",
        "\n",
        "Note: A gamma distribution is a generalization of the exponential distribution. A gamma distribution with shape parameter $α = 1$ and scale parameter $β$ is an  exponential ($β$) distribution. Here, we use a gamma distribution to have more flexibility than we would have had were we to model with an exponential. Rather than returning values between $0$ and $1$, we can return values much larger than $1$ (i.e., the kinds of numbers one would expect to show up in a daily SMS count)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l2QX3nEbofZr"
      },
      "source": [
        "alpha = 1. / 8.\n",
        "\n",
        "λ_1, λ_2 = Gamma(concentration = 1 / alpha, rate = .3).sample(sample_shape = (2,), seed = random.PRNGKey(0))  \n",
        "print(\"λ_1 (randomly taken from Gamma(α) distribution):\\n   \", λ_1)\n",
        "print(\"λ_2 (randomly taken from Gamma(α) distribution):\\n   \", λ_2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uIoKaO4bIAxb"
      },
      "source": [
        "3\\.  For days before $\\tau$, represent the user's received SMS count by sampling from $\\text{Poisson}(\\lambda_1)$, and sample from  $\\text{Poisson}(\\lambda_2)$ for days after $\\tau$. For example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6xxOtwxvpk_P"
      },
      "source": [
        "data = concatenate(\n",
        "    [Poisson(rate = λ_1).sample(sample_shape = τ, seed = random.PRNGKey(0)),\n",
        "     Poisson(rate = λ_2).sample(sample_shape = 80 - τ, seed = random.PRNGKey(0))],\n",
        "    axis = 0)\n",
        "days_range = arange(80)\n",
        "print(\"Artificial day-by-day user SMS count created by sampling:\\n   \", data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5PWuas1oIAxg"
      },
      "source": [
        "4\\. Plot the artificial dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vrGXdyZyIAxh"
      },
      "source": [
        "plt.bar(days_range, data, color = TFColor[3])\n",
        "plt.bar(τ - 1, data[int(τ) - 1], color = 'r', label = \"User behaviour changed\")\n",
        "plt.xlabel(\"Time (days)\")\n",
        "plt.ylabel(\"Count of text-msgs received\")\n",
        "plt.title(\"Artificial dataset\")\n",
        "plt.xlim(0, 80)\n",
        "plt.legend();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ulb46iQIIAxn"
      },
      "source": [
        "It is okay that our fictional dataset does not look like our observed dataset: the probability is incredibly small it indeed would. TFP's engine is designed to find good parameters, $\\lambda_i, \\tau$, that maximise this probability.  \n",
        "\n",
        "\n",
        "The ability to generate an artificial dataset is an interesting side effect of our modeling, and we will see that this ability is a very important method of Bayesian inference. We produce a few more datasets below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vrLFYvVzIAxp"
      },
      "source": [
        "plt.figure(figsize(12.5, 8))\n",
        "for i in range(4):\n",
        "    plt.subplot(4, 1, i + 1)\n",
        "    τ = random.uniform(shape = (1,), \n",
        "                       minval = 0, \n",
        "                       maxval = 80,\n",
        "                       dtype = i32)\n",
        "    alpha = 1. / 8.\n",
        "    λ_1, λ_2 = Gamma(concentration = 1 / alpha, rate = 0.3).sample(sample_shape = (2,)) \n",
        "    data = concatenate([Poisson(rate = λ_1).sample(sample_shape = τ),\n",
        "                           Poisson(rate = λ_2).sample(sample_shape = 80 - τ)],\n",
        "                          axis = 0)\n",
        "    days_range = arange(80)\n",
        "    \n",
        "    plt.bar(days_range, data, color=TFColor[3])\n",
        "    plt.bar(τ - 1, data[int(τ) - 1], color='r', label=\"User behaviour changed\")\n",
        "    plt.xlim(0, 80)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V8QMiJXOIAxv"
      },
      "source": [
        "Later we will see how we use this to make predictions and test the appropriateness of our models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2lU8C4C-IAxw"
      },
      "source": [
        "### Example: Bayesian A/B testing\n",
        "\n",
        "A/B testing is a statistical design for determining the difference of effectiveness between two treatments. For example, a pharmaceutical company is interested in the effectiveness of drug A vs drug B. The company will test drug A on some fraction of their trials, and drug B on the other fraction (this fraction is often 1/2, but we will relax this assumption). After performing enough trials, the in-house statisticians sift through the data to determine which drug yielded better results. \n",
        "\n",
        "Similarly, front-end web developers are interested in which design of their website yields more sales or some other metric of interest. They will route some fraction of visitors to site A, and the other fraction to site B, and record if the visit yielded a sale or not. The data is recorded (in real-time), and analysed afterwards. \n",
        "\n",
        "Often, the post-experiment analysis is done using something called a hypothesis test like *difference of means test* or *difference of proportions test*. This involves often misunderstood quantities like a 'Z-score' and even more confusing 'p-values' (please don't ask). If you have taken a statistics course, you have probably been taught this technique (though not necessarily *learned* this technique). And if you were like me, you may have felt uncomfortable with their derivation -- good: the Bayesian approach to this problem is much more natural. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AFcmkQEyDgyK"
      },
      "source": [
        "### A Simple Case\n",
        "\n",
        "As this is a hacker book, we'll continue with the web-dev example. For the moment, we will focus on the analysis of site A only. Assume that there is some true $0 \\lt p_A \\lt 1$ probability that users who, upon shown site A, eventually purchase from the site. This is the true effectiveness of site A. Currently, this quantity is unknown to us. \n",
        "\n",
        "Suppose site A was shown to $N$ people, and $n$ people purchased from the site. One might conclude hastily that $p_A = \\frac{n}{N}$. Unfortunately, the *observed frequency* $\\frac{n}{N}$ does not necessarily equal $p_A$ -- there is a difference between the *observed frequency* and the *true frequency* of an event. The true frequency can be interpreted as the probability of an event occurring. For example, the true frequency of rolling a 1 on a 6-sided die is $\\frac{1}{6}$. Knowing the true frequency of events like:\n",
        "\n",
        "- fraction of users who make purchases, \n",
        "- frequency of social attributes, \n",
        "- percent of internet users with cats etc. \n",
        "\n",
        "are common requests we ask of Nature. Unfortunately, often Nature hides the true frequency from us and we must *infer* it from observed data.\n",
        "\n",
        "The *observed frequency* is then the frequency we observe: say rolling the die 100 times you may observe 20 rolls of 1. The observed frequency, 0.2, differs from the true frequency, $\\frac{1}{6}$. We can use Bayesian statistics to infer probable values of the true frequency using an appropriate prior and observed data.\n",
        "\n",
        "\n",
        "With respect to our A/B example, we are interested in using what we know, $N$ (the total trials administered) and $n$ (the number of conversions), to estimate what $p_A$, the true frequency of buyers, might be. \n",
        "\n",
        "To setup a Bayesian model, we need to assign prior distributions to our unknown quantities. *A priori*, what do we think $p_A$ might be? For this example, we have no strong conviction about $p_A$, so for now, let's assume $p_A$ is uniform over $[0,1]$:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "blTLKyo2IAxy"
      },
      "source": [
        "# The parameters are the bounds of the Uniform\n",
        "rv_p = Uniform(low = 0., high = 1.)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f0XLF9h3IAx2"
      },
      "source": [
        "Had we had stronger beliefs, we could have expressed them in the prior above.\n",
        "\n",
        "For this example, consider $p_A = 0.05$, and $N = 1500$ users shown site A, and we will simulate whether the user made a purchase or not. To simulate this from $N$ trials, we will use a *Bernoulli* distribution: if  $X\\ \\sim \\text{Ber}(p)$, then $X$ is 1 with probability $p$ and 0 with probability $1 - p$. Of course, in practice we do not know $p_A$, but we will use it here to simulate the data. We can assume then that we can use the following generative model:\n",
        "\n",
        "$$\\begin{align*}\n",
        "p &\\sim \\text{Uniform}(\\text{low}=0,\\text{high}=1) \\\\\n",
        "X\\ &\\sim \\text{Bernoulli}(\\text{prob}=p) \\\\\n",
        "\\text{for }  i &= 1\\ldots N:\\text{# Users}  \\\\\n",
        " X_i\\ &\\sim \\text{Bernoulli}(p_i)\n",
        "\\end{align*}$$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "riLrk5KTIAx4"
      },
      "source": [
        "prob_true = .05  # This is unknown\n",
        "N = 1500\n",
        "\n",
        "# Sample N Bernoulli random variables from Ber(0.05).\n",
        "# each random variable has a 0.05 chance of being a 1.\n",
        "# this is the data-generation step\n",
        "\n",
        "occurrences = Bernoulli(probs = prob_true).sample(sample_shape = N, seed = 10)\n",
        "occurrences_sum = sum(occurrences)\n",
        "occurrences_mean = mean(occurrences.astype(f32))\n",
        "\n",
        "print(f\"Array of {N} Occurences:\", occurrences) \n",
        "# Remember Python treats True == 1, and False == 0\n",
        "print(\"Sum of (True == 1) Occurences:\", occurrences_sum)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UpJrMifMIAx7"
      },
      "source": [
        "The observed frequency is:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "trjtemdNIAx7"
      },
      "source": [
        "# Occurrences.mean is equal to n / N\n",
        "print(f\"What is the observed frequency in Group A? {occurrences_mean:.4f}\")\n",
        "print(f\"Does this equal the true frequency? {occurrences_mean == prob_true}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gue-SRTYIAyA"
      },
      "source": [
        "We can combine our Bernoulli distribution and our observed occurrences into a log probability function based on the two."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ct9o0w7lGaZb"
      },
      "source": [
        "def joint_log_prob(occurrences, prob_A):\n",
        "    \"\"\"Joint log probability optimisation function.\n",
        "        \n",
        "    Args:\n",
        "        occurrences: An array of binary values (0 or 1), representing \n",
        "            the observed frequency\n",
        "        prob_A: scalar estimate of the probability of a 1 appearing \n",
        "    Returns: \n",
        "        sum of the joint log probabilities from all of the prior and conditional distributions\n",
        "    \"\"\"  \n",
        "    rv_prob_A = Uniform(low = 0., high = 1.)\n",
        "    rv_occurrences = Bernoulli(probs = prob_A)\n",
        "    return (\n",
        "          rv_prob_A.log_prob(prob_A)\n",
        "        + sum(rv_occurrences.log_prob(occurrences)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UN7Mh5U-uFye"
      },
      "source": [
        "The goal of probabilistic inference is to find model parameters that may explain\n",
        "data you have observed. TFP performs probabilistic inference by evaluating the\n",
        "model parameters using a `joint_log_prob` function.  The arguments to `joint_log_prob` are data and model parameters—for the model defined in the `joint_log_prob` function itself. The function returns the log of the joint probability that the model parameterised as such generated the observed data per the input arguments.\n",
        "\n",
        "All `joint_log_prob` functions have a common structure:\n",
        "\n",
        "1. The function takes a set of **inputs** to evaluate. Each input is either an\n",
        "observed value or a model parameter.\n",
        "\n",
        "2. The `joint_log_prob` function uses probability distributions to define a **model** for evaluating the inputs. These distributions measure the likelihood of the input values. (By convention, the distribution which measures the likelihood of the variable `foo` will be named `rv_foo` to note that it is a random variable.) We use two types of distributions in `joint_log_prob` functions:\n",
        "\n",
        "  a. **Prior distributions** measure the likelihood of input values.\n",
        "A prior distribution never depends on an input value. Each prior distribution measures the\n",
        "likelihood of a single input value. Each unknown variable—one that has not been\n",
        "observed directly—needs a corresponding prior. Beliefs about which values could\n",
        "be reasonable determine the prior distribution. Choosing a prior can be tricky,\n",
        "so we will cover it in depth in Chapter 6.\n",
        "\n",
        "  b. **Conditional distributions** measure the likelihood of an input value given\n",
        "other input values. Typically, the conditional\n",
        "distributions return the likelihood of observed data given the current guess of parameters in the model, p(observed_data | model_parameters).\n",
        "\n",
        "3. Finally, we calculate and return the **joint log probability** of the inputs.\n",
        "The joint log probability is the sum of the log probabilities from all of the\n",
        "prior and conditional distributions. (We take the sum of log probabilities\n",
        "instead of multiplying the probabilities directly for reasons of numerical\n",
        "stability: floating point numbers in computers cannot represent the very small\n",
        "values necessary to calculate the joint log probability unless they are in \n",
        "log space.) The sum of probabilities is actually an unnormalised density; although the total sum of probabilities  over all possible inputs might not sum to one, the sum of probabilities is proportional to the true probability density. This proportional distribution is sufficient to estimate the distribution of likely inputs.\n",
        "\n",
        "Let's map these terms onto the code above. In this example, the input values\n",
        "are the observed values in `occurrences` and the unknown value for `prob_A`. The `joint_log_prob` takes the current guess for `prob_A`\n",
        "and answers, how likely is the data if `prob_A` is the probability of\n",
        "`occurrences`. The answer depends on two distributions:\n",
        "1. The prior distribution, `rv_prob_A`, indicates how likely the current value of `prob_A` is by itself.\n",
        "2. The conditional distribution, `rv_occurrences`, indicates the likelihood of `occurrences` if `prob_A` were the  probability for the Bernoulli distribution.\n",
        "\n",
        "The sum of the log of these probabilities is the\n",
        "joint log probability. \n",
        "\n",
        "The `joint_log_prob` is particularly useful in conjunction with the [`tfp.mcmc`](https://www.tensorflow.org/probability/api_docs/python/tfp/mcmc)\n",
        "module. Markov chain Monte Carlo (MCMC) algorithms proceed by making educated guesses about the unknown\n",
        "input values and\n",
        "computing what the likelihood of this set of arguments is. (We’ll talk about how it makes those guesses in Chapter 3.) By repeating this process\n",
        "many times, MCMC builds a distribution of likely parameters. Constructing this\n",
        "distribution is the goal of probabilistic inference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rzm3amOgDAGg"
      },
      "source": [
        "Then we run our inference algorithm:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g9XHX0h8IAyB"
      },
      "source": [
        "num_results = 5000\n",
        "num_burnin_steps = 1000\n",
        "num_leapfrog_steps = 2\n",
        "\n",
        "# Set the chain's start state\n",
        "initial_chain_state = [\n",
        "    mean(occurrences.astype(f32)) \n",
        "    * ones((), dtype = f32)]\n",
        "\n",
        "# Since HMC operates over unconstrained space, we need to transform the\n",
        "# samples so they live in real-space.\n",
        "unconstraining_bijectors = [\n",
        "    tfb.Identity()]  # Maps R -> R\n",
        "\n",
        "# Define a closure over our joint_log_prob\n",
        "# The closure makes it so the HMC doesn't try to change the `occurrences` but\n",
        "# instead determines the distributions of other parameters that might generate\n",
        "# the `occurrences` we observed.\n",
        "unnormalised_posterior_log_prob = lambda *args: joint_log_prob(occurrences, *args)\n",
        "\n",
        "# Initialise the step_size\n",
        "step_size = tf.Variable(\n",
        "    initial_value = tf.constant(.5, dtype = f32),\n",
        "    trainable = False,\n",
        "    name = 'step_size')\n",
        "\n",
        "# Define the HMC\n",
        "kernel = tfp.mcmc.TransformedTransitionKernel(\n",
        "    inner_kernel = tfp.mcmc.HamiltonianMonteCarlo(\n",
        "        target_log_prob_fn = unnormalised_posterior_log_prob,\n",
        "        num_leapfrog_steps = num_leapfrog_steps,\n",
        "        step_size = step_size,\n",
        "        state_gradients_are_stopped = True),\n",
        "    bijector = unconstraining_bijectors)\n",
        "\n",
        "kernel = tfp.mcmc.SimpleStepSizeAdaptation(\n",
        "    inner_kernel = kernel,\n",
        "    num_adaptation_steps = int(num_burnin_steps * .8))\n",
        "\n",
        "# Sample from the chain\n",
        "(posterior_prob_A,), kernel_results = tfp.mcmc.sample_chain(\n",
        "    num_results      = num_results,\n",
        "    num_burnin_steps = num_burnin_steps,\n",
        "    current_state    = initial_chain_state,\n",
        "    kernel           = kernel)\n",
        "\n",
        "# print(\"Acceptance rate:\", mean(kernel_results.inner_results.inner_results.is_accepted))\n",
        "\n",
        "burned_prob_A_trace = posterior_prob_A[num_burnin_steps:]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MQUWTY7-LgGv"
      },
      "source": [
        "We plot the posterior distribution of the unknown $p_A$ below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w_P52-CRFJPs"
      },
      "source": [
        "plt.figure(figsize(12.5, 4))\n",
        "plt.title(\"Posterior distribution of $p_A$, the true effectiveness of site A\")\n",
        "plt.vlines(prob_true, 0, 90, linestyle = '--', label = \"True $p_A$ (unknown)\")\n",
        "plt.hist(burned_prob_A_trace, bins = 25, histtype = 'stepfilled')\n",
        "plt.legend();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LdLJ2iriIAyI"
      },
      "source": [
        "Our posterior distribution puts most weight near the true value of $p_A$, but also some weights in the tails. This is a measure of how uncertain we should be, given our observations. Try changing the number of observations, `N`, and observe how the posterior distribution changes.\n",
        "\n",
        "### *A* and *B* Together\n",
        "\n",
        "A similar analysis can be done for site B's response data to determine the analogous $p_B$. But what we are really interested in is the *difference* between $p_A$ and $p_B$. Let's infer $p_A$, $p_B$, *and* $\\text{delta} = p_A - p_B$, all at once. We can do this using TFP's deterministic variables. (We'll assume for this exercise that $p_B = 0.04$, so $\\text{delta} = 0.01$, $N_B = 750$ (significantly less than $N_A$) and we will simulate site B's data like we did for site A's data ). Our model now looks like the following:\n",
        "\n",
        "$$\\begin{align*}\n",
        "p_A &\\sim \\text{Uniform}[\\text{low}=0,\\text{high}=1) \\\\\n",
        "p_B &\\sim \\text{Uniform}[\\text{low}=0,\\text{high}=1) \\\\\n",
        "X\\ &\\sim \\text{Bernoulli}(\\text{prob}=p) \\\\\n",
        "\\text{for }  i &= 1\\ldots N: \\\\\n",
        " X_i\\ &\\sim \\text{Bernoulli}(p_i)\n",
        "\\end{align*}$$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yPDLHl6RIAyJ"
      },
      "source": [
        "# These two quantities are unknown to us\n",
        "true_prob_A = .05\n",
        "true_prob_B = .04\n",
        "\n",
        "# Notice the unequal sample sizes -- no problem in Bayesian analysis\n",
        "N_A = 1500\n",
        "N_B = 750\n",
        "\n",
        "# Generate some observations\n",
        "observations_A = Bernoulli(probs = true_prob_A) \\\n",
        "    .sample(sample_shape = N_A, seed = 6.45)\n",
        "observations_B = Bernoulli(probs = true_prob_B) \\\n",
        "    .sample(sample_shape = N_B, seed = 6.45)\n",
        "\n",
        "print(\"Obs from Site A:\", observations_A[:30], \"...\")\n",
        "print(\"Observed Prob_A:\", mean(observations_A), \"...\")\n",
        "print(\"Obs from Site B:\", observations_B[:30], \"...\")\n",
        "print(\"Observed Prob_B:\", mean(observations_B))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LDzYsDVgMgsz"
      },
      "source": [
        "Below we run inference over the new model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ghHBEdXYtxV"
      },
      "source": [
        "def delta(prob_A, prob_B):\n",
        "    \"\"\"Defining the deterministic delta function. This is our unknown of interest.\n",
        "        \n",
        "    Args:\n",
        "      prob_A: scalar estimate of the probability of a 1 appearing in \n",
        "                observation set A\n",
        "      prob_B: scalar estimate of the probability of a 1 appearing in \n",
        "                observation set B\n",
        "    Returns: \n",
        "      Difference between prob_A and prob_B\n",
        "    \"\"\"\n",
        "    return prob_A - prob_B\n",
        "\n",
        "  \n",
        "def double_joint_log_prob(observations_A, observations_B, \n",
        "                          prob_A, prob_B):\n",
        "    \"\"\"\n",
        "    Joint log probability optimisation function.\n",
        "        \n",
        "    Args:\n",
        "      observations_A: An array of binary values representing the set of \n",
        "                      observations for site A\n",
        "      observations_B: An array of binary values representing the set of \n",
        "                      observations for site B \n",
        "      prob_A: scalar estimate of the probability of a 1 appearing in \n",
        "                observation set A\n",
        "      prob_B: scalar estimate of the probability of a 1 appearing in \n",
        "                observation set B \n",
        "    Returns: \n",
        "      Joint log probability optimisation function.\n",
        "    \"\"\"  \n",
        "    rv_prob_A = Uniform(low = 0., high = 1.)\n",
        "    rv_prob_B = Uniform(low = 0., high = 1.)\n",
        "  \n",
        "    rv_obs_A = Bernoulli(probs = prob_A)\n",
        "    rv_obs_B = Bernoulli(probs = prob_B)\n",
        "  \n",
        "    return (\n",
        "        rv_prob_A.log_prob(prob_A)\n",
        "        + rv_prob_B.log_prob(prob_B)\n",
        "        + tf.reduce_sum(rv_obs_A.log_prob(observations_A))\n",
        "        + tf.reduce_sum(rv_obs_B.log_prob(observations_B)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h0TDeF3IIAyQ"
      },
      "source": [
        "num_results = 5000\n",
        "num_burnin_steps = 1000\n",
        "num_leapfrog_steps = 3\n",
        "\n",
        "# Set the chain's start state\n",
        "initial_chain_state = [    \n",
        "    mean(observations_A.astype(f32)) * ones((), dtype = f32),\n",
        "    mean(observations_B.astype(f32)) * ones((), dtype = f32)]\n",
        "\n",
        "# Since HMC operates over unconstrained space, we need to transform the\n",
        "# samples so they live in real-space.\n",
        "unconstraining_bijectors = [\n",
        "    tfb.Identity(),  # Maps R -> R\n",
        "    tfb.Identity()]  # Maps R -> R\n",
        "\n",
        "# Define a closure over our joint_log_prob\n",
        "unnormalised_posterior_log_prob = lambda *args: double_joint_log_prob(observations_A, observations_B, *args)\n",
        "\n",
        "# Initialise the step_size (it will be automatically adapted)\n",
        "step_size = tf.Variable(\n",
        "    initial_value = tf.constant(.5, dtype = f32),\n",
        "    trainable = False,\n",
        "    name = 'step_size')\n",
        "\n",
        "# Define the HMC\n",
        "kernel = tfp.mcmc.TransformedTransitionKernel(\n",
        "    inner_kernel = tfp.mcmc.HamiltonianMonteCarlo(\n",
        "        target_log_prob_fn = unnormalised_posterior_log_prob,\n",
        "        num_leapfrog_steps = 3,\n",
        "        step_size = step_size,\n",
        "        state_gradients_are_stopped = True),\n",
        "    bijector = unconstraining_bijectors)\n",
        "\n",
        "kernel = tfp.mcmc.SimpleStepSizeAdaptation(\n",
        "    inner_kernel = kernel,\n",
        "    num_adaptation_steps = int(num_burnin_steps * .8))\n",
        "\n",
        "# Sample from the chain\n",
        "(posterior_prob_A, posterior_prob_B), kernel_results = tfp.mcmc.sample_chain(\n",
        "    num_results = num_results,\n",
        "    num_burnin_steps = num_burnin_steps,\n",
        "    current_state = initial_chain_state,\n",
        "    kernel = kernel)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "beUUmGMbdrRr"
      },
      "source": [
        "#### Execute the TF graph to sample from the posterior"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HTYITb9fdqIe"
      },
      "source": [
        "# print(\"Acceptance rate:\", mean(kernel_results.inner_results.inner_results.is_accepted))\n",
        "burned_prob_A_trace = posterior_prob_A[num_burnin_steps:]\n",
        "burned_prob_B_trace = posterior_prob_B[num_burnin_steps:]\n",
        "burned_delta_trace = (posterior_prob_A - posterior_prob_B)[num_burnin_steps:]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YaD67cOkIAyT"
      },
      "source": [
        "Below we plot the posterior distributions for the three unknowns: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PpBXqVKELHRO"
      },
      "source": [
        "plt.figure(figsize(12.5, 12.5))\n",
        "\n",
        "# Histogram of posteriors\n",
        "\n",
        "ax = plt.subplot(311)\n",
        "\n",
        "plt.xlim(0, .1)\n",
        "plt.hist(burned_prob_A_trace, histtype = 'stepfilled', bins = 25, alpha = .85,\n",
        "         label = \"Posterior of $p_A$\", color = TFColor[0])\n",
        "plt.vlines(true_prob_A, 0, 80, linestyle = '--', label = \"True $p_A$ (unknown)\")\n",
        "plt.legend(loc = 'upper right')\n",
        "plt.title(\"Posterior distributions of $p_A$, $p_B$, and delta unknowns\")\n",
        "\n",
        "ax = plt.subplot(312)\n",
        "\n",
        "plt.xlim(0, .1)\n",
        "plt.hist(burned_prob_B_trace, histtype = 'stepfilled', bins = 25, alpha = .85,\n",
        "         label = \"Posterior of $p_B$\", color = TFColor[2])\n",
        "plt.vlines(true_prob_B, 0, 80, linestyle = '--', label = \"True $p_B$ (unknown)\")\n",
        "plt.legend(loc='upper right')\n",
        "\n",
        "ax = plt.subplot(313)\n",
        "plt.hist(burned_delta_trace, histtype = 'stepfilled', bins = 30, alpha = .85,\n",
        "         label = \"Posterior of delta\", color = TFColor[6])\n",
        "plt.vlines(true_prob_A - true_prob_B, 0, 60, linestyle = '--', label = \"True delta (unknown)\")\n",
        "plt.vlines(0, 0, 60, color = 'black', alpha = .2)\n",
        "plt.legend(loc = 'upper right');"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hn-G0eFwIAyh"
      },
      "source": [
        "Notice that as a result of `N_B < N_A`, i.e. we have less data from site B, our posterior distribution of $p_B$ is fatter, implying we are less certain about the true value of $p_B$ than we are of $p_A$.  \n",
        "\n",
        "With respect to the posterior distribution of $\\text{delta}$, we can see that the majority of the distribution is above $\\text{delta}=0$, implying there site A's response is likely better than site B's response. The probability this inference is incorrect is easily computable:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nZxDurxyIAyh"
      },
      "source": [
        "# Count the number of samples less than 0, i.e. the area under the curve\n",
        "# before 0, represent the probability that site A is worse than site B.\n",
        "print(f\"Probability site A is WORSE than site B: {mean(burned_delta_trace < 0):.3f}\")\n",
        "print(f\"Probability site A is BETTER than site B: {mean(burned_delta_trace > 0):.3f}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q8cAEzbUIAyl"
      },
      "source": [
        "If this probability is too high for comfortable decision-making, we can perform more trials on site B (as site B has less samples to begin with, each additional data point for site B contributes more inferential \"power\" than each additional data point for site A). \n",
        "\n",
        "Try playing with the parameters `true_prob_A`, `true_prob_B`, `N_A`, and `N_B`, to see what the posterior of $\\text{delta}$ looks like. Notice in all this, the difference in sample sizes between site A and site B was never mentioned: it naturally fits into Bayesian analysis.\n",
        "\n",
        "I hope the readers feel this style of A/B testing is more natural than hypothesis testing, which has probably confused more than helped practitioners. Later in this book, we will see two extensions of this model: the first to help dynamically adjust for bad sites, and the second will improve the speed of this computation by reducing the analysis to a single equation.   "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-jxOi70IAyl"
      },
      "source": [
        "## An algorithm for human deceit\n",
        "\n",
        "Social data has an additional layer of interest as people are not always honest with responses, which adds a further complication into inference. For example, simply asking individuals \"Have you ever cheated on a test?\" will surely contain some rate of dishonesty. What you can say for certain is that the true rate is less than your observed rate (assuming individuals lie *only* about *not cheating*; I cannot imagine one who would admit \"Yes\" to cheating when in fact they hadn't cheated). \n",
        "\n",
        "To present an elegant solution to circumventing this dishonesty problem, and to demonstrate Bayesian modeling, we first need to introduce the binomial distribution.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qzCqZqzBDpMa"
      },
      "source": [
        "## The Binomial Distribution\n",
        "\n",
        "The binomial distribution is one of the most popular distributions, mostly because of its simplicity and usefulness. Unlike the other distributions we have encountered thus far in the book, the binomial distribution has 2 parameters: $N$, a positive integer representing $N$ trials or number of instances of potential events, and $p$, the probability of an event occurring in a single trial. Like the Poisson distribution, it is a discrete distribution, but unlike the Poisson distribution, it only weighs integers from $0$ to $N$. The mass distribution looks like:\n",
        "\n",
        "$$P( X = k ) =  {{N}\\choose{k}}  p^k(1-p)^{N-k}$$\n",
        "\n",
        "If $X$ is a binomial random variable with parameters $p$ and $N$, denoted $X \\sim \\text{Bin}(N,p)$, then $X$ is the number of events that occurred in the $N$ trials (obviously $0 \\le X \\le N$). The larger $p$ is (while still remaining between 0 and 1), the more events are likely to occur. The expected value of a binomial is equal to $Np$. Below we plot the mass probability distribution for varying parameters. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9I53Ta3maWgJ"
      },
      "source": [
        "N = 10.\n",
        "k_values = arange(0, N + 1, dtype = f32)\n",
        "rv_probs_1 = Binomial(total_count = N, probs = .4).prob(k_values)\n",
        "rv_probs_2 = Binomial(total_count = N, probs = .9).prob(k_values)\n",
        "\n",
        "plt.figure(figsize=(12.5, 4))\n",
        "colors = [TFColor[3], TFColor[0]] \n",
        "\n",
        "plt.bar(k_values - .5, rv_probs_1, color=colors[0],\n",
        "        edgecolor = colors[0],\n",
        "        alpha = .6,\n",
        "        label = f\"$N$: {10.}, $p$: {.4:.1f}\",\n",
        "        linewidth = 3)\n",
        "plt.bar(k_values - .5, rv_probs_2, color=colors[1],\n",
        "        edgecolor = colors[1],\n",
        "        alpha = .6,\n",
        "        label = f\"$N$: {10.}, $p$: {.9:.1f}\",\n",
        "        linewidth = 3)\n",
        "\n",
        "plt.legend(loc = 'upper left')\n",
        "plt.xlim(0, 10.5)\n",
        "plt.xlabel(\"$k$\")\n",
        "plt.ylabel(\"$P(X = k)$\")\n",
        "plt.title(\"Probability mass distributions of binomial random variables\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xT-q-ZI0IAys"
      },
      "source": [
        "The special case when $N = 1$ corresponds to the Bernoulli distribution. There is another connection between Bernoulli and Binomial random variables. If we have $X_1, X_2, ... , X_N$ Bernoulli random variables with the same $p$, then $Z = X_1 + X_2 + ... + X_N \\sim \\text{Binomial}(N, p )$.\n",
        "\n",
        "The expected value of a Bernoulli random variable is $p$. This can be seen by noting the more general Binomial random variable has expected value $Np$ and setting $N=1$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9HmW-50PIAyv"
      },
      "source": [
        "## Example: Cheating among students\n",
        "\n",
        "We will use the binomial distribution to determine the frequency of students cheating during an exam. If we let $N$ be the total number of students who took the exam, and assuming each student is interviewed post-exam (answering without consequence), we will receive integer $X$ \"Yes I did cheat\" answers. We then find the posterior distribution of $p$, given $N$, some specified prior on $p$, and observed data $X$. \n",
        "\n",
        "This is a completely absurd model. No student, even with a free-pass against punishment, would admit to cheating. What we need is a better *algorithm* to ask students if they had cheated. Ideally the algorithm should encourage individuals to be honest while preserving privacy. The following proposed algorithm is a solution I greatly admire for its ingenuity and effectiveness:\n",
        "\n",
        "> In the interview process for each student, the student flips a coin, hidden from the interviewer. The student agrees to answer honestly if the coin comes up heads. Otherwise, if the coin comes up tails, the student (secretly) flips the coin again, and answers \"Yes, I did cheat\" if the coin flip lands heads, and \"No, I did not cheat\", if the coin flip lands tails. This way, the interviewer does not know if a \"Yes\" was the result of a guilty plea, or a Heads on a second coin toss. Thus privacy is preserved and the researchers receive honest answers. \n",
        "\n",
        "I call this the Privacy Algorithm. One could of course argue that the interviewers are still receiving false data since some *Yes*'s are not confessions but instead randomness, but an alternative perspective is that the researchers are discarding approximately half of their original dataset since half of the responses will be noise. But they have gained a systematic data generation process that can be modeled. Furthermore, they do not have to incorporate (perhaps somewhat naively) the possibility of deceitful answers. We can use TFP to dig through this noisy model, and find a posterior distribution for the true frequency of liars. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gPcqaqxMIAyw"
      },
      "source": [
        "Suppose 100 students are being surveyed for cheating, and we wish to find $p$, the proportion of cheaters. There are a few ways we can model this in TFP. I'll demonstrate the most explicit way, and later show a simplified version. Both versions arrive at the same inference. In our data-generation model, we sample $p$, the true proportion of cheaters, from a prior. Since we are quite ignorant about $p$, we will assign it a $\\text{Uniform}(0,1)$ prior."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LIg-xs2LIAyw"
      },
      "source": [
        "N = 100\n",
        "rv_p = Uniform(low = 0., high = 1.)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7L0nMGmrIAy0"
      },
      "source": [
        "Again, thinking of our data-generation model, we assign Bernoulli random variables to the 100 students: 1 implies they cheated and 0 implies they did not. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aXxhrJdtIAy0"
      },
      "source": [
        "N = 100\n",
        "rv_p = Uniform(low = 0., high = 1.)\n",
        "true_answers = Bernoulli(probs = rv_p.sample()).sample(sample_shape = N, seed = 5)\n",
        "\n",
        "print(true_answers)\n",
        "print(tf.reduce_sum(true_answers))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vNB9WGYcIAy4"
      },
      "source": [
        "If we carry out the algorithm, the next step that occurs is the first coin-flip each student makes. This can be modeled again by sampling 100 Bernoulli random variables with $p=1/2$: denote a 1 as a *Heads* and 0 a *Tails*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68t8O39EIAy4"
      },
      "source": [
        "N = 100\n",
        "first_coin_flips = Bernoulli(probs = .5).sample(sample_shape = N, seed = 5)\n",
        "\n",
        "print(first_coin_flips.numpy())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8-ZnScpWIAzA"
      },
      "source": [
        "Although *not everyone* flips a second time, we can still model the possible realisation of second coin-flips:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "acP-4TAfIAzB"
      },
      "source": [
        "N = 100\n",
        "second_coin_flips = Bernoulli(probs = .5).sample(sample_shape = N, seed = 5)\n",
        "\n",
        "print(second_coin_flips)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uiVbAjoTIAzI"
      },
      "source": [
        "Using these variables, we can return a possible realization of the *observed proportion* of \"Yes\" responses. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BJxN0jmBIAzJ"
      },
      "source": [
        "def observed_proportion_calc(t_a = true_answers, \n",
        "                             fc = first_coin_flips,\n",
        "                             sc = second_coin_flips):\n",
        "    \"\"\"Unnormalised log posterior distribution function\n",
        "        \n",
        "    Args:\n",
        "      t_a: array of binary variables representing the true answers\n",
        "      fc: array of binary variables representing the simulated first flips \n",
        "      sc: array of binary variables representing the simulated second flips\n",
        "    Returns: \n",
        "      Observed proportion of coin flips\n",
        "    Closure over: N\n",
        "    \"\"\"\n",
        "    observed = fc * t_a + (1 - fc) * sc\n",
        "    observed_proportion = tf.cast(tf.reduce_sum(observed), f32) / tf.cast(N, f32)\n",
        "    \n",
        "    return tf.cast(observed_proportion, f32)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OoIWHbsNIAzL"
      },
      "source": [
        "The line `fc*t_a + (1-fc)*sc` contains the heart of the Privacy algorithm. Elements in this array are 1 *if and only if* i) the first toss is heads and the student cheated or ii) the first toss is tails, and the second is heads, and are 0 else. Finally, the last line sums this vector and divides by `float(N)`, producing a proportion. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ma5VwRSNIAzM"
      },
      "source": [
        "observed_proportion_val = observed_proportion_calc(t_a = true_answers,\n",
        "                                                   fc = first_coin_flips,\n",
        "                                                   sc = second_coin_flips)\n",
        "\n",
        "print(observed_proportion_val)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HNoBM39rIAzQ"
      },
      "source": [
        "Next we need a dataset. After performing our coin-flipped interviews the researchers received 35 \"Yes\" responses. To put this into a relative perspective, if there truly were no cheaters, we should expect to see on average 1/4 of all responses being a \"Yes\" (half chance of having first coin land Tails, and another half chance of having second coin land Heads), so about 25 responses in a cheat-free world. On the other hand, if *all students cheated*, we should expected to see approximately 3/4 of all responses be \"Yes\". \n",
        "\n",
        "The researchers observe a Binomial random variable, with `N = 100` and `total_yes = 35`:  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SLcH6ZPsIAzR"
      },
      "source": [
        "total_count = 100\n",
        "total_yes = 35"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-kWZd1ygofav"
      },
      "source": [
        "def coin_joint_log_prob(total_yes, total_count, lies_prob):\n",
        "    \"\"\"Joint log probability optimization function.\n",
        "      \n",
        "    Args:\n",
        "      headsflips: Integer for total number of observed heads flips\n",
        "      N: Integer for number of total observation\n",
        "      lies_prob: Test probability of a heads flip (1) for a Binomial distribution\n",
        "    Returns: \n",
        "      Joint log probability optimization function.\n",
        "    \"\"\"\n",
        "  \n",
        "    rv_lies_prob = Uniform(low = 0., high = 1.)\n",
        "\n",
        "    cheated = Bernoulli(probs = tf.cast(lies_prob, f32)).sample(total_count)\n",
        "    first_flips = Bernoulli(probs = .5).sample(total_count)\n",
        "    second_flips = Bernoulli(probs = .5).sample(total_count)\n",
        "    observed_probability = tf.reduce_sum((\n",
        "        cheated * first_flips + (1 - first_flips) * second_flips).astype(f32)) / total_count\n",
        "\n",
        "    rv_yeses = Binomial(total_count = float(total_count),\n",
        "                            probs = observed_probability)\n",
        "    \n",
        "    return (\n",
        "        rv_lies_prob.log_prob(lies_prob)\n",
        "        + tf.reduce_sum(rv_yeses.log_prob(tf.cast(total_yes, f32))))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QZC4TITlIAzV"
      },
      "source": [
        "Below we add all the variables of interest to our Metropolis-Hastings sampler and run our black-box algorithm over the model. It's important to note that we're using a Metropolis-Hastings MCMC instead of a Hamiltonian since we're sampling inside."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Awl3GmgjIAzV"
      },
      "source": [
        "num_steps = 5000\n",
        "num_burnin_steps = 1000\n",
        "total_count = 100\n",
        "\n",
        "# Set the chain's start state.\n",
        "initial_chain_state = [\n",
        "    .4 * ones((), dtype = f32)]\n",
        "\n",
        "# Define a closure over our joint_log_prob.\n",
        "unnormalised_posterior_log_prob = lambda *args: coin_joint_log_prob(total_yes, total_count,  *args)\n",
        "\n",
        "# Define the Metropolis-Hastings\n",
        "# We use a Metropolis-Hastings method here instead of Hamiltonian method\n",
        "# because the coin flips in the above example are non-differentiable and cannot\n",
        "# be used with HMC\n",
        "metropolis = tfp.mcmc.RandomWalkMetropolis(\n",
        "    target_log_prob_fn = unnormalised_posterior_log_prob,\n",
        "    seed = 54)\n",
        "\n",
        "# Sample from the chain.\n",
        "(posterior_p,), kernel_results = tfp.mcmc.sample_chain(\n",
        "    num_results = num_results,\n",
        "    num_burnin_steps = num_burnin_steps,\n",
        "    current_state = initial_chain_state,\n",
        "    kernel = metropolis,\n",
        "    parallel_iterations = 1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lq0OtJDCufOu"
      },
      "source": [
        "##### Executing the TF graph to sample from the posterior"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x--bCsBrr91E"
      },
      "source": [
        "# This cell can take up to 5 minutes in Graph Mode\n",
        "# print(\"Acceptance rate:\", mean(kernel_results.inner_results.inner_results.is_accepted))\n",
        "# print(\"prob_p trace:\", posterior)\n",
        "# print(\"prob_p burned trace:\", posterior_p[num_burnin_steps:])\n",
        "burned_cheating_freq_samples = posterior_p[num_burnin_steps:]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QhCgk98ynq5s"
      },
      "source": [
        "And finally we can plot the results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JoKNmLpxB1yt"
      },
      "source": [
        "plt.figure(figsize(12.5, 6))\n",
        "p_trace = burned_cheating_freq_samples\n",
        "plt.hist(p_trace, histtype = 'stepfilled', density = True, alpha = .85, bins = 30, \n",
        "         label = 'Posterior distribution', color = TFColor[3])\n",
        "plt.vlines([.1, .40], [0, 0], [5, 5], alpha=.3)\n",
        "plt.xlim(0, 1)\n",
        "plt.legend();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tqDMt8xyIAzd"
      },
      "source": [
        "With regards to the above plot, we are still pretty uncertain about what the true frequency of cheaters might be, but we have narrowed it down to a range between 0.1 to 0.4 (marked by the solid lines). This is pretty good, as *a priori* we had no idea how many students might have cheated (hence the uniform distribution for our prior). On the other hand, it is also pretty bad since there is a .3 length window the true value most likely lives in. Have we even gained anything, or are we still too uncertain about the true frequency? \n",
        "\n",
        "I would argue, yes, we have discovered something. It is implausible, according to our posterior, that there are *no cheaters*, i.e. the posterior assigns low probability to $p=0$. Since we started with an uniform prior, treating all values of $p$ as equally plausible, but the data ruled out $p=0$ as a possibility, we can be confident that there were cheaters. \n",
        "\n",
        "This kind of algorithm can be used to gather private information from users and be *reasonably* confident that the data, though noisy, is truthful. \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y0bK5tMAIAze"
      },
      "source": [
        "### Alternative TFP Model\n",
        "\n",
        "Given a value for $p$ (which from our god-like position we know), we can find the probability the student will answer yes: \n",
        "$$\n",
        "\\begin{align}\n",
        "P(\\text{'Yes'}) &= P( \\text{Heads on first coin} )P( \\text{cheater} ) + P( \\text{Tails on first coin} )P( \\text{Heads on second coin} ) \\\\\n",
        "&= \\frac{1}{2}p + \\frac{1}{2}\\frac{1}{2}\\\\\n",
        "&= \\frac{p}{2} + \\frac{1}{4}\n",
        "\\end{align}\n",
        "$$\n",
        "Thus, knowing $p$ we know the probability a student will respond 'Yes'. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m_kgk64TIAzh"
      },
      "source": [
        "If we know the probability of respondents saying \"Yes\", which is `p_skewed`, and we have $N=100$ students, the number of \"Yes\" responses is a binomial random variable with parameters `N` and `p_skewed`.\n",
        "\n",
        "This is where we include our observed 35 \"Yes\" responses out of a total of 100, which are then passed to the `joint_log_prob` in the code section further below, where we define our closure over the`joint_log_prob`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GJ2jFKI7ofa9"
      },
      "source": [
        "N = 100.\n",
        "total_yes = 35.\n",
        "\n",
        "def alt_joint_log_prob(yes_responses, N, prob_cheating):\n",
        "    \"\"\"\n",
        "    Alternative joint log probability optimisation function.\n",
        "        \n",
        "    Args:\n",
        "      yes_responses: Integer for total number of affirmative responses\n",
        "      N: Integer for number of total observation\n",
        "      prob_cheating: Test probability of a student actually cheating\n",
        "    Returns: \n",
        "      Joint log probability optimisation function.\n",
        "    \"\"\"\n",
        "    rv_prob = Uniform(low = 0., high = 1.)\n",
        "    p_skewed = .5 * prob_cheating + .25\n",
        "    rv_yes_responses = Binomial(total_count = tf.cast(N, f32), \n",
        "                                    probs = p_skewed)\n",
        "\n",
        "    return (\n",
        "        rv_prob.log_prob(prob_cheating)\n",
        "        + sum(rv_yes_responses.log_prob(tf.cast(yes_responses, f32)))\n",
        "    )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0clIAcyHIAzj"
      },
      "source": [
        "\n",
        "Below we add all the variables of interest to our HMC component-defining cell and run our black-box algorithm over the model. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zTLIihT_CuLp"
      },
      "source": [
        "num_results = 5000\n",
        "num_burnin_steps = 1000\n",
        "\n",
        "# Set the chain's start state.\n",
        "initial_chain_state = [\n",
        "    .2 * ones((), dtype = f32)]\n",
        "\n",
        "# Since HMC operates over unconstrained space, we need to transform the\n",
        "# samples so they live in real-space.\n",
        "unconstraining_bijectors = [\n",
        "    tfp.bijectors.Sigmoid()]   # Maps [0,1] to R.\n",
        "\n",
        "# Define a closure over our joint_log_prob.\n",
        "# unnormalised_posterior_log_prob = lambda *args: alt_joint_log_prob(headsflips, total_yes, N, *args)\n",
        "unnormalised_posterior_log_prob = lambda *args: alt_joint_log_prob(total_yes, N, *args)\n",
        "\n",
        "# Initialise the step_size (it will be automatically adapted.)\n",
        "step_size = tf.Variable(\n",
        "    initial_value = tf.constant(.5, dtype = f32),\n",
        "    trainable = False,\n",
        "    name = 'skewed_step_size')\n",
        "\n",
        "# Defining the HMC\n",
        "kernel = tfp.mcmc.TransformedTransitionKernel(\n",
        "    inner_kernel = tfp.mcmc.HamiltonianMonteCarlo(\n",
        "        target_log_prob_fn = unnormalised_posterior_log_prob,\n",
        "        num_leapfrog_steps = 2,\n",
        "        step_size = step_size,\n",
        "        step_size_update_fn = tfp.mcmc.make_simple_step_size_update_policy(\n",
        "            num_adaptation_steps = int(num_burnin_steps * .8)),\n",
        "        state_gradients_are_stopped = True),\n",
        "    bijector = unconstraining_bijectors)\n",
        "\n",
        "# Sample from the chain\n",
        "(posterior_skewed_p,), kernel_results = tfp.mcmc.sample_chain(\n",
        "    num_results = num_results,\n",
        "    num_burnin_steps = num_burnin_steps,\n",
        "    current_state = initial_chain_state,\n",
        "    kernel = kernel)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ALvEN1yQkTIx"
      },
      "source": [
        "# This cell may take 5 minutes in Graph Mode\n",
        "# print(\"Acceptance rate:\", mean(kernel_results.inner_results.inner_results.is_accepted))\n",
        "# print(\"Final step size:\", kernel_results.new_step_size[-100:].mean())\n",
        "\n",
        "# print(\"p_skewed trace: \", posterior_skewed_p)\n",
        "# print(\"p_skewed burned trace: \", posterior_skewed_p[num_burnin_steps:])\n",
        "freq_cheating_samples = posterior_skewed_p[num_burnin_steps:]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ye0uC_c-xrWf"
      },
      "source": [
        "Now we can plot our results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_P5Z_uySgi-S"
      },
      "source": [
        "plt.figure(figsize(12.5, 6))\n",
        "p_trace = freq_cheating_samples\n",
        "plt.hist(p_trace, histtype = 'stepfilled', alpha = .85, bins = 30, \n",
        "         label = 'Posterior distribution', color=TFColor[3])\n",
        "plt.vlines([.1, .40], [0, 0], [5, 5], alpha=.2)\n",
        "plt.xlim(0, 1)\n",
        "plt.legend();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lxt6fSRvIAzy"
      },
      "source": [
        "The remainder of this chapter examines some practical examples of TFP and TFP modeling:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KMoiodMmIAzy"
      },
      "source": [
        "## Example: Challenger Space Shuttle Disaster <span id=\"challenger\"/>\n",
        "\n",
        "On January 28, 1986, the twenty-fifth flight of the U.S. space shuttle program ended in disaster when one of the rocket boosters of the Shuttle Challenger exploded shortly after lift-off, killing all seven crew members. The presidential commission on the accident concluded that it was caused by the failure of an O-ring in a field joint on the rocket booster, and that this failure was due to a faulty design that made the O-ring unacceptably sensitive to a number of factors including outside temperature. Of the previous 24 flights, data were available on failures of O-rings on 23, (one was lost at sea), and these data were discussed on the evening preceding the Challenger launch, but unfortunately only the data corresponding to the 7 flights on which there was a damage incident were considered important and these were thought to show no obvious trend. The data are shown below (see [1]):\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tlPZvBWkg5g-"
      },
      "source": [
        "url = 'https://raw.githubusercontent.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/master/Chapter2_MorePyMC/data/challenger_data.csv'\n",
        "filename = wget.download(url)\n",
        "filename"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BNOqG_9zIAzz"
      },
      "source": [
        "plt.figure(figsize(12.5, 3.5))\n",
        "set_printoptions(precision=3, suppress=True)\n",
        "challenger_data = genfromtxt('challenger_data.csv', skip_header=1,\n",
        "                                 usecols=[1, 2], missing_values='NA',\n",
        "                                 delimiter=',')\n",
        "# Drop the NA values\n",
        "challenger_data = challenger_data[~isnan(challenger_data[:, 1])]\n",
        "\n",
        "# Plot it, as a function of tempature (the first column)\n",
        "print(\"Temp (F), O-Ring failure?\")\n",
        "print(challenger_data)\n",
        "\n",
        "plt.scatter(challenger_data[:, 0], challenger_data[:, 1], s=75, color='k',\n",
        "            alpha=.5)\n",
        "plt.yticks([0, 1])\n",
        "plt.ylabel(\"Damage Incident?\")\n",
        "plt.xlabel(\"Outside temperature (Fahrenheit)\")\n",
        "plt.title(\"Defects of the Space Shuttle O-Rings vs temperature\");\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "El9Z_4ulIAz3"
      },
      "source": [
        "It looks clear that *the probability* of damage incidents occurring increases as the outside temperature decreases. We are interested in modeling the probability here because it does not look like there is a strict cutoff point between temperature and a damage incident occurring. The best we can do is ask \"At temperature $t$, what is the probability of a damage incident?\". The goal of this example is to answer that question.\n",
        "\n",
        "We need a function of temperature, call it $p(t)$, that is bounded between 0 and 1 (so as to model a probability) and changes from 1 to 0 as we increase temperature. There are actually many such functions, but the most popular choice is the *logistic function.*\n",
        "\n",
        "$$p(t) = \\frac{1}{ 1 + e^{ \\;\\beta t } } $$\n",
        "\n",
        "In this model, $\\beta$ is the variable we are uncertain about. Below is the function plotted for $\\beta = 1, 3, -5$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U4kW2QIddYEs"
      },
      "source": [
        "def logistic(x, beta):\n",
        "    return 1. / (1. + exp(beta * x))\n",
        "\n",
        "x_vals = linspace(-4., 4., 100)\n",
        "log_beta_1 = logistic(x_vals, 1.)\n",
        "log_beta_3 = logistic(x_vals, 3.)\n",
        "log_beta_m5 = logistic(x_vals, -5.)\n",
        "\n",
        "plt.figure(figsize(12.5, 3))\n",
        "plt.plot(x_vals, log_beta_1, label=r\"$\\beta = 1$\", color=TFColor[0])\n",
        "plt.plot(x_vals, log_beta_3, label=r\"$\\beta = 3$\", color=TFColor[3])\n",
        "plt.plot(x_vals, log_beta_m5, label=r\"$\\beta = -5$\", color=TFColor[6])\n",
        "plt.legend();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H_rdEcFIIAz7"
      },
      "source": [
        "But something is missing. In the plot of the logistic function, the probability changes only near zero, but in our data above the probability changes around 65 to 70. We need to add a *bias* term to our logistic function:\n",
        "\n",
        "$$p(t) = \\frac{1}{ 1 + e^{ \\;\\beta t + \\alpha } } $$\n",
        "\n",
        "Some plots are below, with differing $\\alpha$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T0iZj_eCIAz8"
      },
      "source": [
        "def logistic(x, beta, alpha=0):\n",
        "    \"\"\"Logistic Function with offset\"\"\"\n",
        "    return 1. / (1. + exp((beta * x) + alpha))\n",
        "\n",
        "x_vals = linspace(start = -4., stop = 4., num = 100)\n",
        "log_beta_1_alpha_1 = logistic(x_vals, 1, 1)\n",
        "log_beta_3_alpha_m2 = logistic(x_vals, 3, -2)\n",
        "log_beta_m5_alpha_7 = logistic(x_vals, -5, 7)\n",
        "\n",
        "plt.figure(figsize(12.5, 3))\n",
        "plt.plot(x_vals, log_beta_1, label=r\"$\\beta = 1$\", ls='--', lw=1, color=TFColor[0])\n",
        "plt.plot(x_vals, log_beta_3, label=r\"$\\beta = 3$\", ls='--', lw=1, color=TFColor[3])\n",
        "plt.plot(x_vals, log_beta_m5, label=r\"$\\beta = -5$\", ls='--', lw=1, color=TFColor[6])\n",
        "plt.plot(x_vals, log_beta_1_alpha_1, label=r\"$\\beta = 1, \\alpha = 1$\", color=TFColor[0])\n",
        "plt.plot(x_vals, log_beta_3_alpha_m2, label=r\"$\\beta = 3, \\alpha = -2$\", color=TFColor[3])\n",
        "plt.plot(x_vals, log_beta_m5_alpha_7, label=r\"$\\beta = -5, \\alpha = 7$\", color=TFColor[6])\n",
        "plt.legend(loc='lower left');"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W_B8n8wuIAz9"
      },
      "source": [
        "Adding a constant term $\\alpha$ amounts to shifting the curve left or right (hence why it is called a *bias*).\n",
        "\n",
        "Let's start modeling this in TFP. The $\\beta, \\alpha$ parameters have no reason to be positive, bounded or relatively large, so they are best modeled by a *Normal random variable*, introduced next."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_52Ml-KhIAz9"
      },
      "source": [
        "### Normal distributions\n",
        "\n",
        "A Normal random variable, denoted $X \\sim N(\\mu, 1/\\tau)$, has a distribution with two parameters: the mean, $\\mu$, and the *precision*, $\\tau$. Those familiar with the Normal distribution already have probably seen $\\sigma^2$ instead of $\\tau^{-1}$. They are in fact reciprocals of each other. The change was motivated by simpler mathematical analysis and is an artifact of older Bayesian methods. Just remember: the smaller $\\tau$, the larger the spread of the distribution (i.e. we are more uncertain); the larger $\\tau$, the tighter the distribution (i.e. we are more certain). Regardless, $\\tau$ is always positive. \n",
        "\n",
        "The probability density function of a $N( \\mu, 1/\\tau)$ random variable is:\n",
        "\n",
        "$$ f(x | \\mu, \\tau) = \\sqrt{\\frac{\\tau}{2\\pi}} \\exp\\left( -\\frac{\\tau}{2} (x-\\mu)^2 \\right) $$\n",
        "\n",
        "We plot some different density functions below. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OLw3-8x2hxkm"
      },
      "source": [
        "rand_x_vals = linspace(-8., 7., 150)\n",
        "\n",
        "density_func_1 = Normal(loc = -2., scale = 1. / .7).prob(rand_x_vals)\n",
        "density_func_2 = Normal(loc = 0., scale = 1. / 1).prob(rand_x_vals)\n",
        "density_func_3 = Normal(loc = 3., scale = 1. / 2.8).prob(rand_x_vals)\n",
        "\n",
        "colors = [TFColor[3], TFColor[0], TFColor[6]]\n",
        "\n",
        "plt.figure(figsize(12.5, 3))\n",
        "plt.plot(rand_x_vals, density_func_1,\n",
        "         label=fr'$\\mu = {-2.}, \\tau = {.7:.1f}$', color = TFColor[3])\n",
        "plt.fill_between(rand_x_vals, density_func_1, color = TFColor[3], alpha=.33)\n",
        "plt.plot(rand_x_vals, density_func_2, \n",
        "         label=r\"$\\mu = %d, \\tau = %.1f$\" % (0., 1), color=TFColor[0])\n",
        "plt.fill_between(rand_x_vals, density_func_2, color=TFColor[0], alpha=.33)\n",
        "plt.plot(rand_x_vals, density_func_3,\n",
        "         label=r\"$\\mu = %d, \\tau = %.1f$\" % (3., 2.8), color=TFColor[6])\n",
        "plt.fill_between(rand_x_vals, density_func_3, color=TFColor[6], alpha=.33)\n",
        "\n",
        "plt.legend(loc=r'upper right')\n",
        "plt.xlabel(r\"$x$\")\n",
        "plt.ylabel(r\"Density function at $x$\")\n",
        "plt.title(r\"Probability distribution of three different Normal random variables\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HAxWLKGcIA0A"
      },
      "source": [
        "A Normal random variable can be take on any real number, but the variable is very likely to be relatively close to $\\mu$. In fact, the expected value of a Normal is equal to its $\\mu$ parameter:\n",
        "\n",
        "$$ E[ X | \\mu, \\tau] = \\mu$$\n",
        "\n",
        "and its variance is equal to the inverse of $\\tau$:\n",
        "\n",
        "$$\\text{Var}( X | \\mu, \\tau ) = \\frac{1}{\\tau}$$\n",
        "\n",
        "\n",
        "\n",
        "Below we continue our modeling of the Challenger space craft:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F3DBYxvAIA0B"
      },
      "source": [
        "temperature = challenger_data[:, 0]\n",
        "temperature = tf.convert_to_tensor(temperature, dtype = f32)\n",
        "D = challenger_data[:, 1]  # Defect or not?\n",
        "D = tf.convert_to_tensor(D, dtype = f32)\n",
        "\n",
        "beta = Normal(loc = .3, scale = 1000.).sample()\n",
        "alpha = Normal(loc = -15., scale = 1000.).sample()\n",
        "p_deterministic = Deterministic(loc = 1. / (1. + exp(beta * temperature + alpha))).sample()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PxOWy25CIA0D"
      },
      "source": [
        "We have our probabilities, but how do we connect them to our observed data? A *Bernoulli* random variable with parameter $p$, denoted $\\text{Bernoulli}(p)$, is a random variable that takes value 1 with probability $p$, and 0 else. Thus, our model can look like:\n",
        "\n",
        "$$ \\text{Defect Incident, }D_i \\sim \\text{Bernoulli}( p(t_i) ), \\;\\; i=1..N$$\n",
        "\n",
        "where $p(t)$ is our logistic function and $t_i$ are the temperatures we have observations about. Notice in the code below we set the values of `beta` and `alpha` to 0 in `initial_chain_state`. The reason for this is that if `beta` and `alpha` are very large, they make `p` equal to 1 or 0. Unfortunately, `Bernoulli` does not like probabilities of exactly 0 or 1, though they are mathematically well-defined probabilities. So, by setting the coefficient values to `0`, we set the variable `p` to be a reasonable starting value. This has no effect on our results, nor does it mean we are including any additional information in our prior. It is simply a computational caveat in TFP. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vRqoyxqnofbT"
      },
      "source": [
        "def challenger_joint_log_prob(D, temperature, alpha, beta):\n",
        "    \"\"\"Joint log probability optimization function.\n",
        "        \n",
        "    Args:\n",
        "      D: The Data from the challenger disaster representing presence or \n",
        "         absence of defect\n",
        "      temperature_: The Data from the challenger disaster, specifically the temperature on \n",
        "         the days of the observation of the presence or absence of a defect\n",
        "      alpha: one of the inputs of the HMC\n",
        "      beta: one of the inputs of the HMC\n",
        "    Returns: \n",
        "      Joint log probability optimization function.\n",
        "    \"\"\"\n",
        "    rv_alpha = Normal(loc = 0., scale = 1000.)\n",
        "    rv_beta = Normal(loc = 0., scale = 1000.)\n",
        "\n",
        "    # Make this into a logit\n",
        "    logistic_p = 1. / (1. + exp(beta * tf.cast(temperature, f32) + alpha))\n",
        "    rv_observed = Bernoulli(probs = logistic_p)\n",
        "    \n",
        "    return (\n",
        "        rv_alpha.log_prob(alpha)\n",
        "        + rv_beta.log_prob(beta)\n",
        "        + sum(rv_observed.log_prob(D)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oHU-MbPxs8iL",
        "cellView": "code"
      },
      "source": [
        "num_results = 5000\n",
        "num_burnin_steps = 1000\n",
        "\n",
        "# Set the chain's start state.\n",
        "initial_chain_state = [\n",
        "    0. * ones((), dtype = f32),\n",
        "    0. * ones((), dtype = f32)]\n",
        "\n",
        "# Since HMC operates over unconstrained space, we need to transform the\n",
        "# samples so they live in real-space.\n",
        "# Alpha is 100x of beta approximately, so apply Affine scalar bijector\n",
        "# to multiply the unconstrained alpha by 100 to get back to \n",
        "# the Challenger problem space\n",
        "unconstraining_bijectors = [\n",
        "    tfb.AffineScalar(100.),\n",
        "    tfb.Identity()]\n",
        "\n",
        "# Define a closure over our joint_log_prob.\n",
        "unnormalised_posterior_log_prob = lambda *args: challenger_joint_log_prob(D, temperature, *args)\n",
        "\n",
        "# Initialise the step_size. (It will be automatically adapted.)\n",
        "step_size = tf.Variable(\n",
        "    initial_value = tf.constant(.01, dtype = f32),\n",
        "    trainable = False,\n",
        "    name = 'step_size')\n",
        "\n",
        "# Define the HMC\n",
        "kernel = tfp.mcmc.TransformedTransitionKernel(\n",
        "    inner_kernel = tfp.mcmc.HamiltonianMonteCarlo(\n",
        "        target_log_prob_fn = unnormalised_posterior_log_prob,\n",
        "        num_leapfrog_steps = 40,  # To improve convergence\n",
        "        step_size = step_size,\n",
        "        state_gradients_are_stopped = True),\n",
        "    bijector = unconstraining_bijectors)\n",
        "\n",
        "kernel = tfp.mcmc.SimpleStepSizeAdaptation(\n",
        "    inner_kernel = kernel,\n",
        "    num_adaptation_steps = int(num_burnin_steps * .8))\n",
        "\n",
        "# Sample from the chain\n",
        "(posterior_alpha, posterior_beta), kernel_results = tfp.mcmc.sample_chain(\n",
        "    num_results = num_results,\n",
        "    num_burnin_steps = num_burnin_steps,\n",
        "    current_state = initial_chain_state,\n",
        "    kernel = kernel)\n",
        "\n",
        "# print(\"Acceptance rate:\", mean(kernel_results.inner_results.inner_results.is_accepted))\n",
        "# print(\"Final step size:\", kernel_results.new_step_size[-100:].mean())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xIGyBkilIA0G"
      },
      "source": [
        "We have trained our model on the observed data, so lets look at the posterior distributions for $\\alpha$ and $\\beta$:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pdgjgw9RiluO"
      },
      "source": [
        "plt.figure(figsize(12.5, 6))\n",
        "plt.subplot(211)\n",
        "plt.title(r\"Posterior distributions of the variables $\\alpha, \\beta$\")\n",
        "plt.hist(posterior_beta, histtype='stepfilled', bins=35, alpha=.85,\n",
        "         label=r\"Posterior of $\\beta$\", color=TFColor[6], density=True)\n",
        "plt.legend()\n",
        "plt.subplot(212)\n",
        "plt.hist(posterior_alpha, histtype='stepfilled', bins=35, alpha=.85,\n",
        "         label=r\"Posterior of $\\alpha$\", color=TFColor[0], density=True)\n",
        "plt.legend();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gp0QmuZvIA0L"
      },
      "source": [
        "All samples of $\\beta$ are greater than 0. If instead the posterior was centered around 0, we may suspect that $\\beta = 0$, implying that temperature has no effect on the probability of defect. \n",
        "\n",
        "Similarly, all $\\alpha$ posterior values are negative and far away from 0, implying that it is correct to believe that $\\alpha$ is significantly less than 0. \n",
        "\n",
        "Regarding the spread of the data, we are very uncertain about what the true parameters might be (though considering the low sample size and the large overlap of defects-to-nondefects this behaviour is perhaps expected).  \n",
        "\n",
        "Next, let's look at the *expected probability* for a specific value of the temperature. That is, we average over all samples from the posterior to get a likely value for $p(t_i)$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EIzyJL_3IA0P"
      },
      "source": [
        "alpha_samples_1d = posterior_alpha[:, None]  # Best to make them 1d\n",
        "beta_samples_1d = posterior_beta[:, None]\n",
        "\n",
        "beta_mean = mean(transpose(beta_samples_1d)[0])\n",
        "alpha_mean = mean(transpose(alpha_samples_1d)[0])\n",
        "\n",
        "print(\"beta mean:\", beta_mean)\n",
        "print(\"alpha mean:\", alpha_mean)\n",
        "\n",
        "def logistic(x, beta, alpha=0):\n",
        "    \"\"\"Logistic function with alpha and beta.\"\"\"\n",
        "    return 1. / (1. + exp((beta * x) + alpha))\n",
        "\n",
        "t = linspace(start = tf.reduce_min(temperature) - 5,\n",
        "                stop = tf.reduce_max(temperature) + 5,\n",
        "                num = 2500)[:, None]\n",
        "p_t = logistic(transpose(t), beta_samples_1d, alpha_samples_1d)\n",
        "mean_prob_t = logistic(transpose(t), beta_mean, alpha_mean)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ri4BriJHPJNg"
      },
      "source": [
        "plt.figure(figsize(12.5, 3))\n",
        "plt.plot(t, transpose(mean_prob_t), lw=3, label=\"Average posterior \\nprobability of defect\")\n",
        "plt.plot(t, transpose(p_t)[:, 0], ls='--', label=\"Realization from posterior\")\n",
        "plt.plot(t, transpose(p_t)[:, -8], ls='--', label=\"Realization from posterior\")\n",
        "plt.scatter(temperature, D, color='k', s=50, alpha=.5)\n",
        "plt.title(\"Posterior expected value of probability of defect; plus realizations\")\n",
        "plt.legend(loc='lower left')\n",
        "plt.ylim(-.1, 1.1)\n",
        "plt.xlim(tf.reduce_min(t), tf.reduce_max(t))\n",
        "plt.ylabel(\"Probability\")\n",
        "plt.xlabel(\"Temperature\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iI7Fosv1IA0T"
      },
      "source": [
        "Above we also plotted two possible realizations of what the actual underlying system might be. Both are equally likely as any other draw. The blue line is what occurs when we average all the 20000 possible dotted lines together.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k_rZvVwz7zHI"
      },
      "source": [
        "from scipy.stats.mstats import mquantiles\n",
        "\n",
        "# Vectorised bottom and top 2.5% quantiles for \"confidence interval\"\n",
        "qs = mquantiles(p_t, [.025, .975], axis=0)\n",
        "plt.fill_between(t[:, 0], *qs, alpha=.7, color='#7A68A6')\n",
        "plt.plot(t[:, 0], qs[0], label=\"95% CI\", color='#7A68A6', alpha=.7)\n",
        "plt.plot(t[:, 0], mean_prob_t[0, :], lw=1, ls='--', color='k',\n",
        "         label=\"Average posterior \\nprobability of defect\")\n",
        "plt.xlim(tf.reduce_min(t), tf.reduce_max(t))\n",
        "plt.ylim(-.02, 1.02)\n",
        "plt.legend(loc='lower left')\n",
        "plt.scatter(temperature, D, color='k', s=50, alpha=.5)\n",
        "plt.xlabel(\"Temp, $t$\")\n",
        "plt.ylabel(\"Probability estimate\")\n",
        "plt.title(\"Posterior probability estimates given temp. $t$\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rtcc9pBFIA0X"
      },
      "source": [
        "The 95% credible interval, or 95% CI, painted in purple, represents the interval, for each temperature, that contains 95% of the distribution. For example, at 65 degrees, we can be 95% sure that the probability of defect lies between 0.25 and 0.85.\n",
        "\n",
        "More generally, we can see that as the temperature nears 60 degrees, the CI's spread out over $[0,1]$ quickly. As we pass 70 degrees, the CI's tighten again. This can give us insight about how to proceed next: we should probably test more O-rings around 60-65 temperature to get a better estimate of probabilities in that range. Similarly, when reporting to scientists your estimates, you should be very cautious about simply telling them the expected probability, as we can see this does not reflect how *wide* the posterior distribution is."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lesq_3oIIA0Y"
      },
      "source": [
        "### What about the day of the Challenger disaster?\n",
        "\n",
        "On the day of the Challenger disaster, the outside temperature was 31 degrees Fahrenheit. What is the posterior distribution of a defect occurring,  given this temperature? The distribution is plotted below. It looks almost guaranteed that the Challenger was going to be subject to defective O-rings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AYbamYmdUdBZ"
      },
      "source": [
        "plt.figure(figsize(12.5, 3))\n",
        "prob_31 = logistic(31, posterior_beta, posterior_alpha)\n",
        "plt.xlim(.98, 1)\n",
        "plt.hist(prob_31, bins=10, density=True, histtype='stepfilled')\n",
        "plt.title(\"Posterior distribution of probability of defect, given $t = 31$\")\n",
        "plt.xlabel(\"probability of defect occurring in O-ring\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WjAFZ8W9IA0c"
      },
      "source": [
        "### Is our model appropriate?\n",
        "\n",
        "The skeptical reader will say \"You deliberately chose the logistic function for $p(t)$ and the specific priors. Perhaps other functions or priors will give different results. How do I know I have chosen a good model?\" This is absolutely true. To consider an extreme situation, what if I had chosen the function $p(t) = 1,\\; \\forall t$, which guarantees a defect always occurring: I would have again predicted disaster on January 28th. Yet this is clearly a poorly chosen model. On the other hand, if I did choose the logistic function for $p(t)$, but specified all my priors to be very tight around 0, likely we would have very different posterior distributions. How do we know our model is an expression of the data? This encourages us to measure the model's **goodness of fit**.\n",
        "\n",
        "We can think: *how can we test whether our model is a bad fit?* An idea is to compare observed data with artificial dataset which we can simulate. The rationale is that if the simulated dataset does not appear similar, statistically, to the observed dataset, then likely our model is not accurately represented the observed data. \n",
        "\n",
        "Previously in this Chapter, we simulated an artificial dataset for the SMS example. To do this, we sampled values from the priors. We saw how varied the resulting datasets looked like, and rarely did they mimic our observed dataset. In the current example,  we should sample from the *posterior* distributions to create *very plausible datasets*. Luckily, our Bayesian framework makes this very easy. We only need to gather samples from the distribution of choice, and specify the number of samples, the shape of the samples (we had 21 observations in our original dataset, so we'll make the shape of each sample 21), and the probability we want to use to determine the ratio of 1 observations to 0 observations.\n",
        "\n",
        "\n",
        "Hence we create the following:\n",
        "\n",
        "```python\n",
        "simulated_data = Bernoulli(probs = p).sample(sample_shape = (N,))\n",
        "```\n",
        "Let's simulate 10,000:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MvFwyz9hwROg"
      },
      "source": [
        "alpha = alpha_mean  # Basing these values on the outputs of the model above\n",
        "beta = beta_mean\n",
        "p_deterministic = Deterministic(\n",
        "    loc = 1. / (1. + exp(beta * temperature + alpha))).sample()  # seed=6.45\n",
        "simulated_data = Bernoulli(probs = p_deterministic).sample(sample_shape = (10000,))\n",
        "\n",
        "simulations = simulated_data\n",
        "print(\"Number of simulations:            \", simulations.shape[0])\n",
        "print(\"Number data points per simulation:\", simulations.shape[1])\n",
        "\n",
        "plt.figure(figsize(12.5, 12))\n",
        "plt.title(\"Simulated dataset using posterior parameters\")\n",
        "for i in range(4):\n",
        "    ax = plt.subplot(4, 1, i + 1)\n",
        "    plt.scatter(temperature, simulations[1000 * i, :], color='k',\n",
        "                s=50, alpha=.6)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RFSaMzNQIA0l"
      },
      "source": [
        "Note that the above plots are different (if you can think of a cleaner way to present this, please send a pull request and answer [here](http://stats.stackexchange.com/questions/53078/how-to-visualize-bayesian-goodness-of-fit-for-logistic-regression)!).\n",
        "\n",
        "We wish to assess how good our model is. 'Good' is a subjective term of course, so results must be relative to other models. \n",
        "\n",
        "We will be doing this graphically as well, which may seem like an even less objective method. The alternative is to use *Bayesian p-values*. These are still subjective, as the proper cutoff between good and bad is arbitrary. Gelman emphasises that the graphical tests are more illuminating [3] than p-value tests. We agree.\n",
        "\n",
        "The following graphical test is a novel data-viz approach to logistic regression. The plots are called *separation plots*[4]. For a suite of models we wish to compare, each model is plotted on an individual separation plot. I leave most of the technical details about separation plots to the very accessible [original paper](http://mdwardlab.com/sites/default/files/GreenhillWardSacks.pdf), but I'll summarise their use here.\n",
        "\n",
        "For each model, we calculate the proportion of times the posterior simulation proposed a value of 1 for a particular temperature, i.e. compute $P( \\;\\text{Defect} = 1 | t, \\alpha, \\beta )$ by averaging. This gives us the posterior probability of a defect at each data point in our dataset. For example, for the model we used above:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3ho1cPLAIA0l"
      },
      "source": [
        "posterior_probability = mean(simulations, axis = 0)\n",
        "print(\"Posterior prob of defect | Realised defect \")\n",
        "for i in range(len(D)):\n",
        "    print(\"%.2f                     |   %d\" % (posterior_probability[i], D[i]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-q4yysOiIA0n"
      },
      "source": [
        "Next we sort each column by the posterior probabilities:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i3TkXUhSIA0n"
      },
      "source": [
        "ix = argsort(posterior_probability)\n",
        "print(\"probb | defect \")\n",
        "for i in range(len(D)):\n",
        "    print(\"%.2f  |   %d\" % (posterior_probability[ix[i]], D[ix[i]]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ajvopQADIA0p"
      },
      "source": [
        "We can present the above data better in a figure: we've created a `separation_plot` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tFR1_yu8IA0p"
      },
      "source": [
        " def separation_plot(p, y, **kwargs):\n",
        "    \"\"\"This function creates a separation plot for logistic and probit classification. \n",
        "    See http://mdwardlab.com/sites/default/files/GreenhillWardSacks.pdf\n",
        "    \n",
        "    p: tf.Tensor, the proportions/probabilities, can be a nxM matrix which represents M models.\n",
        "    y: tf.Tensor, the 0-1 response variables.\n",
        "    \"\"\"    \n",
        "    assert p.shape[0] == y.shape[0]\n",
        "    n = p.shape[0]\n",
        "\n",
        "    if len(p.shape) > 1:\n",
        "        m = p.shape[1]\n",
        "    else:\n",
        "        p = tf.reshape(p, (n, 1))\n",
        "        m = p.shape[1]\n",
        "\n",
        "    colors = ['#EEE', '#348ABD']\n",
        "\n",
        "    fig = plt.figure()\n",
        "    \n",
        "    for i in range(m):\n",
        "        ax = fig.add_subplot(m, 1, i + 1)\n",
        "        ix = argsort(p[:, i])\n",
        "        # Plot the different bars\n",
        "        bars = ax.bar(arange(n), ones(n), width=1.,\n",
        "                      color=colors[i%2],  # colors_bmh[ y[ix].astype(int) ]\n",
        "                      edgecolor='none')\n",
        "        ax.plot(arange(n + 1), append(p[ix, i], p[ix, i][-1]), 'k',\n",
        "                linewidth=1., drawstyle='steps-post')\n",
        "        # Create expected value bar.\n",
        "        ax.vlines([tf.reduce_sum((1 - p[ix, i]))], [0], [1])\n",
        "        plt.xlim(0, n)\n",
        "        \n",
        "    plt.tight_layout()\n",
        "    return\n",
        "\n",
        "plt.figure(figsize(11., 3))\n",
        "separation_plot(posterior_probability, D)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Si1VVcbLuICA"
      },
      "source": [
        "# TODO Check those types\n",
        "posterior_probability, D  # tf.Tensor(shape=(23,), dtype=int32), f.Tensor(shape=(23,), dtype=float32)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tBuY2lSaIA0s"
      },
      "source": [
        "The snaking-line is the sorted probabilities, blue bars denote defects, and empty space (or grey bars for the optimistic readers) denote non-defects. As the probability rises, we see more and more defects occur. On the right hand side, the plot suggests that as the posterior probability is large (line close to 1), then more defects are realised. This is good behaviour. Ideally, all the blue bars *should* be close to the right-hand side, and deviations from this reflect missed predictions. \n",
        "\n",
        "The black vertical line is the expected number of defects we should observe, given this model. This allows the user to see how the total number of events predicted by the model compares to the actual number of events in the data.\n",
        "\n",
        "It is much more informative to compare this to separation plots for other models. Below we compare our model (top) versus three others:\n",
        "\n",
        "1. the perfect model, which predicts the posterior probability to be equal 1 if a defect did occur.\n",
        "2. a completely random model, which predicts random probabilities regardless of temperature.\n",
        "3. a constant model:  where $P(D = 1 \\; | \\; t) = c, \\;\\; \\forall t$. The best choice for $c$ is the observed frequency of defects, in this case 7/23.  \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RbX1nHrBIA0s"
      },
      "source": [
        "plt.figure(figsize(11., 2))\n",
        "\n",
        "# Our temperature-dependent model\n",
        "separation_plot(posterior_probability, D)\n",
        "plt.title(\"Temperature-dependent model\")\n",
        "\n",
        "# Perfect model\n",
        "# i.e. the probability of defect is equal to if a defect occurred or not.\n",
        "p = D\n",
        "separation_plot(p, D)\n",
        "plt.title(\"Perfect model\")\n",
        "\n",
        "# Random predictions\n",
        "p = np.random.rand(23)\n",
        "separation_plot(p, D)\n",
        "plt.title(\"Random model\")\n",
        "\n",
        "# Constant model\n",
        "constant_prob = 7. / 23 * ones(23)\n",
        "separation_plot(constant_prob, D)\n",
        "plt.title(\"Constant-prediction model\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "etHuMg8OIA0u"
      },
      "source": [
        "In the random model, we can see that as the probability increases there is no clustering of defects to the right-hand side. Similarly for the constant model.\n",
        "\n",
        "In the perfect model, the probability line is not well shown, as it is stuck to the bottom and top of the figure. Of course the perfect model is only for demonstration, and we cannot infer any scientific inference from it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_I_3h7lsIA0v"
      },
      "source": [
        "## Exercises\n",
        "\n",
        "1\\. Try putting in extreme values for our observations in the cheating example. What happens if we observe 25 affirmative responses? 10? 50? "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_B-K0Neyx7pZ"
      },
      "source": [
        "# type your code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ulECHYwMIA0v"
      },
      "source": [
        "2\\. Try plotting $\\alpha$ samples versus $\\beta$ samples.  Why might the resulting plot look like this?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "21v-EuHZIA0v"
      },
      "source": [
        "# type your code here\n",
        "plt.figure(figsize(12.5, 4))\n",
        "\n",
        "plt.scatter(alpha, beta, alpha=.1)\n",
        "plt.title(\"Why does the plot look like this?\")\n",
        "plt.xlabel(r\"$\\alpha$\")\n",
        "plt.ylabel(r\"$\\beta$\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6RYS1_jIA0y"
      },
      "source": [
        "## References\n",
        "\n",
        "[1] Dalal, Fowlkes and Hoadley (1989),JASA, 84, 945-957.\n",
        "\n",
        "[2] Cronin, Beau. \"Why Probabilistic Programming Matters.\" 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/+BeauCronin/posts/KpeRdJKR6Z1>.\n",
        "\n",
        "[3] Gelman, Andrew. \"Philosophy and the practice of Bayesian statistics.\" British Journal of Mathematical and Statistical Psychology. (2012): n. page. Web. 2 Apr. 2013.\n",
        "\n",
        "[4] Greenhill, Brian, Michael D. Ward, and Audrey Sacks. \"The Separation Plot: A New Visual Method for Evaluating the Fit of Binary Models.\" American Journal of Political Science. 55.No.4 (2011): n. page. Web. 2 Apr. 2013.\n"
      ]
    }
  ]
}