BasicStatisticalTesting_json.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "BasicStatisticalTesting.json",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kwangnam/006697/blob/master/BasicStatisticalTesting_json.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z4uVbJpuDLio"
      },
      "source": [
        "In this lecture we're going to review some of the basics of statistical testing in python. We're going to\n",
        "talk about hypothesis testing, statistical significance, and using scipy to run student's t-tests."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-L36jnM3DLis"
      },
      "source": [
        "# We use statistics in a lot of different ways in data science, and on this lecture, I want to refresh your\n",
        "# knowledge of hypothesis testing, which is a core data analysis activity behind experimentation. The goal of\n",
        "# hypothesis testing is to determine if, for instance, the two different conditions we have in an experiment \n",
        "# have resulted in different impacts\n",
        "\n",
        "# Let's import our usual numpy and pandas libraries\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "# Now let's bring in some new libraries from scipy\n",
        "from scipy import stats"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vC_Veo3DDLit"
      },
      "source": [
        "# Now, scipy is an interesting collection of libraries for data science and you'll use most or perpahs all of\n",
        "# these libraries. It includes numpy and pandas, but also plotting libraries such as matplotlib, and a\n",
        "# number of scientific library functions as well"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "j64FiRVqDLit",
        "outputId": "c1e52e2c-f982-4f98-e346-c0659d248df1"
      },
      "source": [
        "# When we do hypothesis testing, we actually have two statements of interest: the first is our actual\n",
        "# explanation, which we call the alternative hypothesis, and the second is that the explanation we have is not\n",
        "# sufficient, and we call this the null hypothesis. Our actual testing method is to determine whether the null\n",
        "# hypothesis is true or not. If we find that there is a difference between groups, then we can reject the null\n",
        "# hypothesis and we accept our alternative.\n",
        "\n",
        "# Let's see an example of this; we're going to use some grade data\n",
        "df=pd.read_csv ('datasets/grades.csv')\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>student_id</th>\n",
              "      <th>assignment1_grade</th>\n",
              "      <th>assignment1_submission</th>\n",
              "      <th>assignment2_grade</th>\n",
              "      <th>assignment2_submission</th>\n",
              "      <th>assignment3_grade</th>\n",
              "      <th>assignment3_submission</th>\n",
              "      <th>assignment4_grade</th>\n",
              "      <th>assignment4_submission</th>\n",
              "      <th>assignment5_grade</th>\n",
              "      <th>assignment5_submission</th>\n",
              "      <th>assignment6_grade</th>\n",
              "      <th>assignment6_submission</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>B73F2C11-70F0-E37D-8B10-1D20AFED50B1</td>\n",
              "      <td>92.733946</td>\n",
              "      <td>2015-11-02 06:55:34.282000000</td>\n",
              "      <td>83.030552</td>\n",
              "      <td>2015-11-09 02:22:58.938000000</td>\n",
              "      <td>67.164441</td>\n",
              "      <td>2015-11-12 08:58:33.998000000</td>\n",
              "      <td>53.011553</td>\n",
              "      <td>2015-11-16 01:21:24.663000000</td>\n",
              "      <td>47.710398</td>\n",
              "      <td>2015-11-20 13:24:59.692000000</td>\n",
              "      <td>38.168318</td>\n",
              "      <td>2015-11-22 18:31:15.934000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>98A0FAE0-A19A-13D2-4BB5-CFBFD94031D1</td>\n",
              "      <td>86.790821</td>\n",
              "      <td>2015-11-29 14:57:44.429000000</td>\n",
              "      <td>86.290821</td>\n",
              "      <td>2015-12-06 17:41:18.449000000</td>\n",
              "      <td>69.772657</td>\n",
              "      <td>2015-12-10 08:54:55.904000000</td>\n",
              "      <td>55.098125</td>\n",
              "      <td>2015-12-13 17:32:30.941000000</td>\n",
              "      <td>49.588313</td>\n",
              "      <td>2015-12-19 23:26:39.285000000</td>\n",
              "      <td>44.629482</td>\n",
              "      <td>2015-12-21 17:07:24.275000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>D0F62040-CEB0-904C-F563-2F8620916C4E</td>\n",
              "      <td>85.512541</td>\n",
              "      <td>2016-01-09 05:36:02.389000000</td>\n",
              "      <td>85.512541</td>\n",
              "      <td>2016-01-09 06:39:44.416000000</td>\n",
              "      <td>68.410033</td>\n",
              "      <td>2016-01-15 20:22:45.882000000</td>\n",
              "      <td>54.728026</td>\n",
              "      <td>2016-01-11 12:41:50.749000000</td>\n",
              "      <td>49.255224</td>\n",
              "      <td>2016-01-11 17:31:12.489000000</td>\n",
              "      <td>44.329701</td>\n",
              "      <td>2016-01-17 16:24:42.765000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>FFDF2B2C-F514-EF7F-6538-A6A53518E9DC</td>\n",
              "      <td>86.030665</td>\n",
              "      <td>2016-04-30 06:50:39.801000000</td>\n",
              "      <td>68.824532</td>\n",
              "      <td>2016-04-30 17:20:38.727000000</td>\n",
              "      <td>61.942079</td>\n",
              "      <td>2016-05-12 07:47:16.326000000</td>\n",
              "      <td>49.553663</td>\n",
              "      <td>2016-05-07 16:09:20.485000000</td>\n",
              "      <td>49.553663</td>\n",
              "      <td>2016-05-24 12:51:18.016000000</td>\n",
              "      <td>44.598297</td>\n",
              "      <td>2016-05-26 08:09:12.058000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5ECBEEB6-F1CE-80AE-3164-E45E99473FB4</td>\n",
              "      <td>64.813800</td>\n",
              "      <td>2015-12-13 17:06:10.750000000</td>\n",
              "      <td>51.491040</td>\n",
              "      <td>2015-12-14 12:25:12.056000000</td>\n",
              "      <td>41.932832</td>\n",
              "      <td>2015-12-29 14:25:22.594000000</td>\n",
              "      <td>36.929549</td>\n",
              "      <td>2015-12-28 01:29:55.901000000</td>\n",
              "      <td>33.236594</td>\n",
              "      <td>2015-12-29 14:46:06.628000000</td>\n",
              "      <td>33.236594</td>\n",
              "      <td>2016-01-05 01:06:59.546000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                             student_id  assignment1_grade  \\\n",
              "0  B73F2C11-70F0-E37D-8B10-1D20AFED50B1          92.733946   \n",
              "1  98A0FAE0-A19A-13D2-4BB5-CFBFD94031D1          86.790821   \n",
              "2  D0F62040-CEB0-904C-F563-2F8620916C4E          85.512541   \n",
              "3  FFDF2B2C-F514-EF7F-6538-A6A53518E9DC          86.030665   \n",
              "4  5ECBEEB6-F1CE-80AE-3164-E45E99473FB4          64.813800   \n",
              "\n",
              "          assignment1_submission  assignment2_grade  \\\n",
              "0  2015-11-02 06:55:34.282000000          83.030552   \n",
              "1  2015-11-29 14:57:44.429000000          86.290821   \n",
              "2  2016-01-09 05:36:02.389000000          85.512541   \n",
              "3  2016-04-30 06:50:39.801000000          68.824532   \n",
              "4  2015-12-13 17:06:10.750000000          51.491040   \n",
              "\n",
              "          assignment2_submission  assignment3_grade  \\\n",
              "0  2015-11-09 02:22:58.938000000          67.164441   \n",
              "1  2015-12-06 17:41:18.449000000          69.772657   \n",
              "2  2016-01-09 06:39:44.416000000          68.410033   \n",
              "3  2016-04-30 17:20:38.727000000          61.942079   \n",
              "4  2015-12-14 12:25:12.056000000          41.932832   \n",
              "\n",
              "          assignment3_submission  assignment4_grade  \\\n",
              "0  2015-11-12 08:58:33.998000000          53.011553   \n",
              "1  2015-12-10 08:54:55.904000000          55.098125   \n",
              "2  2016-01-15 20:22:45.882000000          54.728026   \n",
              "3  2016-05-12 07:47:16.326000000          49.553663   \n",
              "4  2015-12-29 14:25:22.594000000          36.929549   \n",
              "\n",
              "          assignment4_submission  assignment5_grade  \\\n",
              "0  2015-11-16 01:21:24.663000000          47.710398   \n",
              "1  2015-12-13 17:32:30.941000000          49.588313   \n",
              "2  2016-01-11 12:41:50.749000000          49.255224   \n",
              "3  2016-05-07 16:09:20.485000000          49.553663   \n",
              "4  2015-12-28 01:29:55.901000000          33.236594   \n",
              "\n",
              "          assignment5_submission  assignment6_grade  \\\n",
              "0  2015-11-20 13:24:59.692000000          38.168318   \n",
              "1  2015-12-19 23:26:39.285000000          44.629482   \n",
              "2  2016-01-11 17:31:12.489000000          44.329701   \n",
              "3  2016-05-24 12:51:18.016000000          44.598297   \n",
              "4  2015-12-29 14:46:06.628000000          33.236594   \n",
              "\n",
              "          assignment6_submission  \n",
              "0  2015-11-22 18:31:15.934000000  \n",
              "1  2015-12-21 17:07:24.275000000  \n",
              "2  2016-01-17 16:24:42.765000000  \n",
              "3  2016-05-26 08:09:12.058000000  \n",
              "4  2016-01-05 01:06:59.546000000  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IAhiZ1SpDLiu",
        "outputId": "78c8a1f8-2d1f-4d17-a29b-43df5423b03c"
      },
      "source": [
        "# If we take a look at the data frame inside, we see we have six different assignments. Lets look at some\n",
        "# summary statistics for this DataFrame\n",
        "print(\"There are {} rows and {} columns\".format(df.shape[0], df.shape[1]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "There are 2315 rows and 13 columns\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "IhRyFquIDLiv",
        "outputId": "e9960db7-ef51-422f-badb-fc4a90694f8f"
      },
      "source": [
        "# For the purpose of this lecture, let's segment this population into two pieces. Let's say those who finish\n",
        "# the first assignment by the end of December 2015, we'll call them early finishers, and those who finish it \n",
        "# sometime after that, we'll call them late finishers.\n",
        "\n",
        "early_finishers=df[pd.to_datetime(df['assignment1_submission']) < '2016']\n",
        "early_finishers.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>student_id</th>\n",
              "      <th>assignment1_grade</th>\n",
              "      <th>assignment1_submission</th>\n",
              "      <th>assignment2_grade</th>\n",
              "      <th>assignment2_submission</th>\n",
              "      <th>assignment3_grade</th>\n",
              "      <th>assignment3_submission</th>\n",
              "      <th>assignment4_grade</th>\n",
              "      <th>assignment4_submission</th>\n",
              "      <th>assignment5_grade</th>\n",
              "      <th>assignment5_submission</th>\n",
              "      <th>assignment6_grade</th>\n",
              "      <th>assignment6_submission</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>B73F2C11-70F0-E37D-8B10-1D20AFED50B1</td>\n",
              "      <td>92.733946</td>\n",
              "      <td>2015-11-02 06:55:34.282000000</td>\n",
              "      <td>83.030552</td>\n",
              "      <td>2015-11-09 02:22:58.938000000</td>\n",
              "      <td>67.164441</td>\n",
              "      <td>2015-11-12 08:58:33.998000000</td>\n",
              "      <td>53.011553</td>\n",
              "      <td>2015-11-16 01:21:24.663000000</td>\n",
              "      <td>47.710398</td>\n",
              "      <td>2015-11-20 13:24:59.692000000</td>\n",
              "      <td>38.168318</td>\n",
              "      <td>2015-11-22 18:31:15.934000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>98A0FAE0-A19A-13D2-4BB5-CFBFD94031D1</td>\n",
              "      <td>86.790821</td>\n",
              "      <td>2015-11-29 14:57:44.429000000</td>\n",
              "      <td>86.290821</td>\n",
              "      <td>2015-12-06 17:41:18.449000000</td>\n",
              "      <td>69.772657</td>\n",
              "      <td>2015-12-10 08:54:55.904000000</td>\n",
              "      <td>55.098125</td>\n",
              "      <td>2015-12-13 17:32:30.941000000</td>\n",
              "      <td>49.588313</td>\n",
              "      <td>2015-12-19 23:26:39.285000000</td>\n",
              "      <td>44.629482</td>\n",
              "      <td>2015-12-21 17:07:24.275000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5ECBEEB6-F1CE-80AE-3164-E45E99473FB4</td>\n",
              "      <td>64.813800</td>\n",
              "      <td>2015-12-13 17:06:10.750000000</td>\n",
              "      <td>51.491040</td>\n",
              "      <td>2015-12-14 12:25:12.056000000</td>\n",
              "      <td>41.932832</td>\n",
              "      <td>2015-12-29 14:25:22.594000000</td>\n",
              "      <td>36.929549</td>\n",
              "      <td>2015-12-28 01:29:55.901000000</td>\n",
              "      <td>33.236594</td>\n",
              "      <td>2015-12-29 14:46:06.628000000</td>\n",
              "      <td>33.236594</td>\n",
              "      <td>2016-01-05 01:06:59.546000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>D09000A0-827B-C0FF-3433-BF8FF286E15B</td>\n",
              "      <td>71.647278</td>\n",
              "      <td>2015-12-28 04:35:32.836000000</td>\n",
              "      <td>64.052550</td>\n",
              "      <td>2016-01-03 21:05:38.392000000</td>\n",
              "      <td>64.752550</td>\n",
              "      <td>2016-01-07 08:55:43.692000000</td>\n",
              "      <td>57.467295</td>\n",
              "      <td>2016-01-11 00:45:28.706000000</td>\n",
              "      <td>57.467295</td>\n",
              "      <td>2016-01-11 00:54:13.579000000</td>\n",
              "      <td>57.467295</td>\n",
              "      <td>2016-01-20 19:54:46.166000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>C9D51293-BD58-F113-4167-A7C0BAFCB6E5</td>\n",
              "      <td>66.595568</td>\n",
              "      <td>2015-12-25 02:29:28.415000000</td>\n",
              "      <td>52.916454</td>\n",
              "      <td>2015-12-31 01:42:30.046000000</td>\n",
              "      <td>48.344809</td>\n",
              "      <td>2016-01-05 23:34:02.180000000</td>\n",
              "      <td>47.444809</td>\n",
              "      <td>2016-01-02 07:48:42.517000000</td>\n",
              "      <td>37.955847</td>\n",
              "      <td>2016-01-03 21:27:04.266000000</td>\n",
              "      <td>37.955847</td>\n",
              "      <td>2016-01-19 15:24:31.060000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                             student_id  assignment1_grade  \\\n",
              "0  B73F2C11-70F0-E37D-8B10-1D20AFED50B1          92.733946   \n",
              "1  98A0FAE0-A19A-13D2-4BB5-CFBFD94031D1          86.790821   \n",
              "4  5ECBEEB6-F1CE-80AE-3164-E45E99473FB4          64.813800   \n",
              "5  D09000A0-827B-C0FF-3433-BF8FF286E15B          71.647278   \n",
              "8  C9D51293-BD58-F113-4167-A7C0BAFCB6E5          66.595568   \n",
              "\n",
              "          assignment1_submission  assignment2_grade  \\\n",
              "0  2015-11-02 06:55:34.282000000          83.030552   \n",
              "1  2015-11-29 14:57:44.429000000          86.290821   \n",
              "4  2015-12-13 17:06:10.750000000          51.491040   \n",
              "5  2015-12-28 04:35:32.836000000          64.052550   \n",
              "8  2015-12-25 02:29:28.415000000          52.916454   \n",
              "\n",
              "          assignment2_submission  assignment3_grade  \\\n",
              "0  2015-11-09 02:22:58.938000000          67.164441   \n",
              "1  2015-12-06 17:41:18.449000000          69.772657   \n",
              "4  2015-12-14 12:25:12.056000000          41.932832   \n",
              "5  2016-01-03 21:05:38.392000000          64.752550   \n",
              "8  2015-12-31 01:42:30.046000000          48.344809   \n",
              "\n",
              "          assignment3_submission  assignment4_grade  \\\n",
              "0  2015-11-12 08:58:33.998000000          53.011553   \n",
              "1  2015-12-10 08:54:55.904000000          55.098125   \n",
              "4  2015-12-29 14:25:22.594000000          36.929549   \n",
              "5  2016-01-07 08:55:43.692000000          57.467295   \n",
              "8  2016-01-05 23:34:02.180000000          47.444809   \n",
              "\n",
              "          assignment4_submission  assignment5_grade  \\\n",
              "0  2015-11-16 01:21:24.663000000          47.710398   \n",
              "1  2015-12-13 17:32:30.941000000          49.588313   \n",
              "4  2015-12-28 01:29:55.901000000          33.236594   \n",
              "5  2016-01-11 00:45:28.706000000          57.467295   \n",
              "8  2016-01-02 07:48:42.517000000          37.955847   \n",
              "\n",
              "          assignment5_submission  assignment6_grade  \\\n",
              "0  2015-11-20 13:24:59.692000000          38.168318   \n",
              "1  2015-12-19 23:26:39.285000000          44.629482   \n",
              "4  2015-12-29 14:46:06.628000000          33.236594   \n",
              "5  2016-01-11 00:54:13.579000000          57.467295   \n",
              "8  2016-01-03 21:27:04.266000000          37.955847   \n",
              "\n",
              "          assignment6_submission  \n",
              "0  2015-11-22 18:31:15.934000000  \n",
              "1  2015-12-21 17:07:24.275000000  \n",
              "4  2016-01-05 01:06:59.546000000  \n",
              "5  2016-01-20 19:54:46.166000000  \n",
              "8  2016-01-19 15:24:31.060000000  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-nskv-ibDLiv"
      },
      "source": [
        "# So, you have lots of skills now with pandas, how would you go about getting the late_finishers dataframe?\n",
        "# Why don't you pause the video and give it a try."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ACq2HUrZDLiw",
        "outputId": "c1a18882-8fc3-41ad-fd88-a4f909cfad8e"
      },
      "source": [
        "# Here's my solution. First, the dataframe df and the early_finishers share index values, so I really just\n",
        "# want everything in the df which is not in early_finishers\n",
        "late_finishers=df[~df.index.isin(early_finishers.index)]\n",
        "late_finishers.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>student_id</th>\n",
              "      <th>assignment1_grade</th>\n",
              "      <th>assignment1_submission</th>\n",
              "      <th>assignment2_grade</th>\n",
              "      <th>assignment2_submission</th>\n",
              "      <th>assignment3_grade</th>\n",
              "      <th>assignment3_submission</th>\n",
              "      <th>assignment4_grade</th>\n",
              "      <th>assignment4_submission</th>\n",
              "      <th>assignment5_grade</th>\n",
              "      <th>assignment5_submission</th>\n",
              "      <th>assignment6_grade</th>\n",
              "      <th>assignment6_submission</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>D0F62040-CEB0-904C-F563-2F8620916C4E</td>\n",
              "      <td>85.512541</td>\n",
              "      <td>2016-01-09 05:36:02.389000000</td>\n",
              "      <td>85.512541</td>\n",
              "      <td>2016-01-09 06:39:44.416000000</td>\n",
              "      <td>68.410033</td>\n",
              "      <td>2016-01-15 20:22:45.882000000</td>\n",
              "      <td>54.728026</td>\n",
              "      <td>2016-01-11 12:41:50.749000000</td>\n",
              "      <td>49.255224</td>\n",
              "      <td>2016-01-11 17:31:12.489000000</td>\n",
              "      <td>44.329701</td>\n",
              "      <td>2016-01-17 16:24:42.765000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>FFDF2B2C-F514-EF7F-6538-A6A53518E9DC</td>\n",
              "      <td>86.030665</td>\n",
              "      <td>2016-04-30 06:50:39.801000000</td>\n",
              "      <td>68.824532</td>\n",
              "      <td>2016-04-30 17:20:38.727000000</td>\n",
              "      <td>61.942079</td>\n",
              "      <td>2016-05-12 07:47:16.326000000</td>\n",
              "      <td>49.553663</td>\n",
              "      <td>2016-05-07 16:09:20.485000000</td>\n",
              "      <td>49.553663</td>\n",
              "      <td>2016-05-24 12:51:18.016000000</td>\n",
              "      <td>44.598297</td>\n",
              "      <td>2016-05-26 08:09:12.058000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>3217BE3F-E4B0-C3B6-9F64-462456819CE4</td>\n",
              "      <td>87.498744</td>\n",
              "      <td>2016-03-05 11:05:25.408000000</td>\n",
              "      <td>69.998995</td>\n",
              "      <td>2016-03-09 07:29:52.405000000</td>\n",
              "      <td>55.999196</td>\n",
              "      <td>2016-03-16 22:31:24.316000000</td>\n",
              "      <td>50.399276</td>\n",
              "      <td>2016-03-18 07:19:26.032000000</td>\n",
              "      <td>45.359349</td>\n",
              "      <td>2016-03-19 10:35:41.869000000</td>\n",
              "      <td>45.359349</td>\n",
              "      <td>2016-03-23 14:02:00.987000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>F1CB5AA1-B3DE-5460-FAFF-BE951FD38B5F</td>\n",
              "      <td>80.576090</td>\n",
              "      <td>2016-01-24 18:24:25.619000000</td>\n",
              "      <td>72.518481</td>\n",
              "      <td>2016-01-27 13:37:12.943000000</td>\n",
              "      <td>65.266633</td>\n",
              "      <td>2016-01-30 14:34:36.581000000</td>\n",
              "      <td>65.266633</td>\n",
              "      <td>2016-02-03 22:08:49.002000000</td>\n",
              "      <td>65.266633</td>\n",
              "      <td>2016-02-16 14:22:23.664000000</td>\n",
              "      <td>65.266633</td>\n",
              "      <td>2016-02-18 08:35:04.796000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>E2C617C2-4654-622C-AB50-1550C4BE42A0</td>\n",
              "      <td>59.270882</td>\n",
              "      <td>2016-03-06 12:06:26.185000000</td>\n",
              "      <td>59.270882</td>\n",
              "      <td>2016-03-13 02:07:25.289000000</td>\n",
              "      <td>53.343794</td>\n",
              "      <td>2016-03-17 07:30:09.241000000</td>\n",
              "      <td>53.343794</td>\n",
              "      <td>2016-03-20 21:45:56.229000000</td>\n",
              "      <td>42.675035</td>\n",
              "      <td>2016-03-27 15:55:04.414000000</td>\n",
              "      <td>38.407532</td>\n",
              "      <td>2016-03-30 20:33:13.554000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                             student_id  assignment1_grade  \\\n",
              "2  D0F62040-CEB0-904C-F563-2F8620916C4E          85.512541   \n",
              "3  FFDF2B2C-F514-EF7F-6538-A6A53518E9DC          86.030665   \n",
              "6  3217BE3F-E4B0-C3B6-9F64-462456819CE4          87.498744   \n",
              "7  F1CB5AA1-B3DE-5460-FAFF-BE951FD38B5F          80.576090   \n",
              "9  E2C617C2-4654-622C-AB50-1550C4BE42A0          59.270882   \n",
              "\n",
              "          assignment1_submission  assignment2_grade  \\\n",
              "2  2016-01-09 05:36:02.389000000          85.512541   \n",
              "3  2016-04-30 06:50:39.801000000          68.824532   \n",
              "6  2016-03-05 11:05:25.408000000          69.998995   \n",
              "7  2016-01-24 18:24:25.619000000          72.518481   \n",
              "9  2016-03-06 12:06:26.185000000          59.270882   \n",
              "\n",
              "          assignment2_submission  assignment3_grade  \\\n",
              "2  2016-01-09 06:39:44.416000000          68.410033   \n",
              "3  2016-04-30 17:20:38.727000000          61.942079   \n",
              "6  2016-03-09 07:29:52.405000000          55.999196   \n",
              "7  2016-01-27 13:37:12.943000000          65.266633   \n",
              "9  2016-03-13 02:07:25.289000000          53.343794   \n",
              "\n",
              "          assignment3_submission  assignment4_grade  \\\n",
              "2  2016-01-15 20:22:45.882000000          54.728026   \n",
              "3  2016-05-12 07:47:16.326000000          49.553663   \n",
              "6  2016-03-16 22:31:24.316000000          50.399276   \n",
              "7  2016-01-30 14:34:36.581000000          65.266633   \n",
              "9  2016-03-17 07:30:09.241000000          53.343794   \n",
              "\n",
              "          assignment4_submission  assignment5_grade  \\\n",
              "2  2016-01-11 12:41:50.749000000          49.255224   \n",
              "3  2016-05-07 16:09:20.485000000          49.553663   \n",
              "6  2016-03-18 07:19:26.032000000          45.359349   \n",
              "7  2016-02-03 22:08:49.002000000          65.266633   \n",
              "9  2016-03-20 21:45:56.229000000          42.675035   \n",
              "\n",
              "          assignment5_submission  assignment6_grade  \\\n",
              "2  2016-01-11 17:31:12.489000000          44.329701   \n",
              "3  2016-05-24 12:51:18.016000000          44.598297   \n",
              "6  2016-03-19 10:35:41.869000000          45.359349   \n",
              "7  2016-02-16 14:22:23.664000000          65.266633   \n",
              "9  2016-03-27 15:55:04.414000000          38.407532   \n",
              "\n",
              "          assignment6_submission  \n",
              "2  2016-01-17 16:24:42.765000000  \n",
              "3  2016-05-26 08:09:12.058000000  \n",
              "6  2016-03-23 14:02:00.987000000  \n",
              "7  2016-02-18 08:35:04.796000000  \n",
              "9  2016-03-30 20:33:13.554000000  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Q_rNsAtDLiw"
      },
      "source": [
        "# There are lots of other ways to do this. For instance, you could just copy and paste the first projection\n",
        "# and change the sign from less than to greater than or equal to. This is ok, but if you decide you want to\n",
        "# change the date down the road you have to remember to change it in two places. You could also do a join of\n",
        "# the dataframe df with early_finishers - if you do a left join you only keep the items in the left dataframe,\n",
        "# so this would have been a good answer. You also could have written a function that determines if someone is\n",
        "# early or late, and then called .apply() on the dataframe and added a new column to the dataframe. This is a\n",
        "# pretty reasonable answer as well."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TS2BOqgUDLix",
        "outputId": "650cdb9e-bab1-4977-b9f1-2eaab8a28cf0"
      },
      "source": [
        "# As you've seen, the pandas data frame object has a variety of statistical functions associated with it. If\n",
        "# we call the mean function directly on the data frame, we see that each of the means for the assignments are\n",
        "# calculated. Let's compare the means for our two populations\n",
        "\n",
        "print(early_finishers['assignment1_grade'].mean())\n",
        "print(late_finishers['assignment1_grade'].mean())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "74.94728457024303\n",
            "74.0450648477065\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w05rytX4DLix",
        "outputId": "9e337250-5511-45ba-a1b1-8c724af2f337"
      },
      "source": [
        "# Ok, these look pretty similar. But, are they the same? What do we mean by similar? This is where the\n",
        "# students' t-test comes in. It allows us to form the alternative hypothesis (\"These are different\") as well\n",
        "# as the null hypothesis (\"These are the same\") and then test that null hypothesis.\n",
        "\n",
        "# When doing hypothesis testing, we have to choose a significance level as a threshold for how much of a\n",
        "# chance we're willing to accept. This significance level is typically called alpha. #For this example, let's\n",
        "# use a threshold of 0.05 for our alpha or 5%. Now this is a commonly used number but it's really quite\n",
        "# arbitrary.\n",
        "\n",
        "# The SciPy library contains a number of different statistical tests and forms a basis for hypothesis testing\n",
        "# in Python and we're going to use the ttest_ind() function which does an independent t-test (meaning the\n",
        "# populations are not related to one another). The result of ttest_index() are the t-statistic and a p-value.\n",
        "# It's this latter value, the probability, which is most important to us, as it indicates the chance (between\n",
        "# 0 and 1) of our null hypothesis being True.\n",
        "\n",
        "# Let's bring in our ttest_ind function\n",
        "from scipy.stats import ttest_ind\n",
        "\n",
        "# Let's run this function with our two populations, looking at the assignment 1 grades\n",
        "ttest_ind(early_finishers['assignment1_grade'], late_finishers['assignment1_grade'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Ttest_indResult(statistic=1.322354085372139, pvalue=0.1861810110171455)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hhNZQXBfDLiy"
      },
      "source": [
        "# So here we see that the probability is 0.18, and this is above our alpha value of 0.05. This means that we\n",
        "# cannot reject the null hypothesis. The null hypothesis was that the two populations are the same, and we\n",
        "# don't have enough certainty in our evidence (because it is greater than alpha) to come to a conclusion to\n",
        "# the contrary. This doesn't mean that we have proven the populations are the same."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TbQTzYP8DLiy",
        "outputId": "a0f113b9-2ba0-4f98-91a8-4dbaed750f2c"
      },
      "source": [
        "# Why don't we check the other assignment grades?\n",
        "print(ttest_ind(early_finishers['assignment2_grade'], late_finishers['assignment2_grade']))\n",
        "print(ttest_ind(early_finishers['assignment3_grade'], late_finishers['assignment3_grade']))\n",
        "print(ttest_ind(early_finishers['assignment4_grade'], late_finishers['assignment4_grade']))\n",
        "print(ttest_ind(early_finishers['assignment5_grade'], late_finishers['assignment5_grade']))\n",
        "print(ttest_ind(early_finishers['assignment6_grade'], late_finishers['assignment6_grade']))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Ttest_indResult(statistic=1.2514717608216366, pvalue=0.2108889627004424)\n",
            "Ttest_indResult(statistic=1.6133726558705392, pvalue=0.10679998102227865)\n",
            "Ttest_indResult(statistic=0.049671157386456125, pvalue=0.960388729789337)\n",
            "Ttest_indResult(statistic=-0.05279315545404755, pvalue=0.9579012739746492)\n",
            "Ttest_indResult(statistic=-0.11609743352612056, pvalue=0.9075854011989656)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oKVKc0suDLiz"
      },
      "source": [
        "# Ok, so it looks like in this data we do not have enough evidence to suggest the populations differ with\n",
        "# respect to grade. Let's take a look at those p-values for a moment though, because they are saying things\n",
        "# that can inform experimental design down the road. For instance, one of the assignments, assignment 3, has a\n",
        "# p-value around 0.1. This means that if we accepted a level of chance similarity of 11% this would have been\n",
        "# considered statistically significant. As a research, this would suggest to me that there is something here\n",
        "# worth considering following up on. For instance, if we had a small number of participants (we don't) or if\n",
        "# there was something unique about this assignment as it relates to our experiment (whatever it was) then\n",
        "# there may be followup experiments we could run."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tSo0dGBaDLi0",
        "outputId": "222c8b39-bcba-4a73-a432-0706c6f14b41"
      },
      "source": [
        "# P-values have come under fire recently for being insuficient for telling us enough about the interactions\n",
        "# which are happening, and two other techniques, confidence intervalues and bayesian analyses, are being used\n",
        "# more regularly. One issue with p-values is that as you run more tests you are likely to get a value which\n",
        "# is statistically significant just by chance.\n",
        "\n",
        "# Lets see a simulation of this. First, lets create a data frame of 100 columns, each with 100 numbers\n",
        "df1=pd.DataFrame([np.random.random(100) for x in range(100)])\n",
        "df1.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>...</th>\n",
              "      <th>90</th>\n",
              "      <th>91</th>\n",
              "      <th>92</th>\n",
              "      <th>93</th>\n",
              "      <th>94</th>\n",
              "      <th>95</th>\n",
              "      <th>96</th>\n",
              "      <th>97</th>\n",
              "      <th>98</th>\n",
              "      <th>99</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.249058</td>\n",
              "      <td>0.748065</td>\n",
              "      <td>0.245871</td>\n",
              "      <td>0.164790</td>\n",
              "      <td>0.321661</td>\n",
              "      <td>0.355034</td>\n",
              "      <td>0.882641</td>\n",
              "      <td>0.304377</td>\n",
              "      <td>0.409110</td>\n",
              "      <td>0.080049</td>\n",
              "      <td>...</td>\n",
              "      <td>0.900937</td>\n",
              "      <td>0.740939</td>\n",
              "      <td>0.353839</td>\n",
              "      <td>0.928663</td>\n",
              "      <td>0.011866</td>\n",
              "      <td>0.876878</td>\n",
              "      <td>0.609879</td>\n",
              "      <td>0.721572</td>\n",
              "      <td>0.589627</td>\n",
              "      <td>0.554882</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.731003</td>\n",
              "      <td>0.592047</td>\n",
              "      <td>0.488109</td>\n",
              "      <td>0.969236</td>\n",
              "      <td>0.873551</td>\n",
              "      <td>0.976036</td>\n",
              "      <td>0.296337</td>\n",
              "      <td>0.398915</td>\n",
              "      <td>0.258550</td>\n",
              "      <td>0.925547</td>\n",
              "      <td>...</td>\n",
              "      <td>0.478983</td>\n",
              "      <td>0.242443</td>\n",
              "      <td>0.225005</td>\n",
              "      <td>0.803914</td>\n",
              "      <td>0.576076</td>\n",
              "      <td>0.190479</td>\n",
              "      <td>0.696137</td>\n",
              "      <td>0.573191</td>\n",
              "      <td>0.505417</td>\n",
              "      <td>0.626049</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.315190</td>\n",
              "      <td>0.081075</td>\n",
              "      <td>0.326315</td>\n",
              "      <td>0.166250</td>\n",
              "      <td>0.257618</td>\n",
              "      <td>0.406736</td>\n",
              "      <td>0.996209</td>\n",
              "      <td>0.986399</td>\n",
              "      <td>0.339623</td>\n",
              "      <td>0.828867</td>\n",
              "      <td>...</td>\n",
              "      <td>0.157851</td>\n",
              "      <td>0.496227</td>\n",
              "      <td>0.983652</td>\n",
              "      <td>0.971251</td>\n",
              "      <td>0.986353</td>\n",
              "      <td>0.547430</td>\n",
              "      <td>0.255769</td>\n",
              "      <td>0.165918</td>\n",
              "      <td>0.242719</td>\n",
              "      <td>0.575260</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.512193</td>\n",
              "      <td>0.639647</td>\n",
              "      <td>0.241200</td>\n",
              "      <td>0.819644</td>\n",
              "      <td>0.439495</td>\n",
              "      <td>0.111223</td>\n",
              "      <td>0.982239</td>\n",
              "      <td>0.574194</td>\n",
              "      <td>0.935731</td>\n",
              "      <td>0.609363</td>\n",
              "      <td>...</td>\n",
              "      <td>0.373206</td>\n",
              "      <td>0.271515</td>\n",
              "      <td>0.942992</td>\n",
              "      <td>0.701573</td>\n",
              "      <td>0.039384</td>\n",
              "      <td>0.085479</td>\n",
              "      <td>0.284002</td>\n",
              "      <td>0.509064</td>\n",
              "      <td>0.112163</td>\n",
              "      <td>0.334849</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.506768</td>\n",
              "      <td>0.799246</td>\n",
              "      <td>0.233428</td>\n",
              "      <td>0.849028</td>\n",
              "      <td>0.682260</td>\n",
              "      <td>0.904570</td>\n",
              "      <td>0.962683</td>\n",
              "      <td>0.734952</td>\n",
              "      <td>0.933580</td>\n",
              "      <td>0.904864</td>\n",
              "      <td>...</td>\n",
              "      <td>0.263317</td>\n",
              "      <td>0.895435</td>\n",
              "      <td>0.327082</td>\n",
              "      <td>0.872009</td>\n",
              "      <td>0.566034</td>\n",
              "      <td>0.474804</td>\n",
              "      <td>0.258897</td>\n",
              "      <td>0.316417</td>\n",
              "      <td>0.433785</td>\n",
              "      <td>0.127601</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 100 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "         0         1         2         3         4         5         6   \\\n",
              "0  0.249058  0.748065  0.245871  0.164790  0.321661  0.355034  0.882641   \n",
              "1  0.731003  0.592047  0.488109  0.969236  0.873551  0.976036  0.296337   \n",
              "2  0.315190  0.081075  0.326315  0.166250  0.257618  0.406736  0.996209   \n",
              "3  0.512193  0.639647  0.241200  0.819644  0.439495  0.111223  0.982239   \n",
              "4  0.506768  0.799246  0.233428  0.849028  0.682260  0.904570  0.962683   \n",
              "\n",
              "         7         8         9   ...        90        91        92        93  \\\n",
              "0  0.304377  0.409110  0.080049  ...  0.900937  0.740939  0.353839  0.928663   \n",
              "1  0.398915  0.258550  0.925547  ...  0.478983  0.242443  0.225005  0.803914   \n",
              "2  0.986399  0.339623  0.828867  ...  0.157851  0.496227  0.983652  0.971251   \n",
              "3  0.574194  0.935731  0.609363  ...  0.373206  0.271515  0.942992  0.701573   \n",
              "4  0.734952  0.933580  0.904864  ...  0.263317  0.895435  0.327082  0.872009   \n",
              "\n",
              "         94        95        96        97        98        99  \n",
              "0  0.011866  0.876878  0.609879  0.721572  0.589627  0.554882  \n",
              "1  0.576076  0.190479  0.696137  0.573191  0.505417  0.626049  \n",
              "2  0.986353  0.547430  0.255769  0.165918  0.242719  0.575260  \n",
              "3  0.039384  0.085479  0.284002  0.509064  0.112163  0.334849  \n",
              "4  0.566034  0.474804  0.258897  0.316417  0.433785  0.127601  \n",
              "\n",
              "[5 rows x 100 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gRtye0l8DLi0"
      },
      "source": [
        "# Pause this and reflect -- do you understand the list comprehension and how I created this DataFrame? You\n",
        "# don't have to use a list comprehension to do this, but you should be able to read this and figure out how it\n",
        "# works as this is a commonly used approach on web forums."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eIJ2M7lGDLi0"
      },
      "source": [
        "# Ok, let's create a second dataframe\n",
        "df2=pd.DataFrame([np.random.random(100) for x in range(100)])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nxfZMGZ-DLi1",
        "outputId": "6e86e30e-fa20-43ae-9001-e8fd477b521b"
      },
      "source": [
        "# Are these two DataFrames the same? Maybe a better question is, for a given row inside of df1, is it the same\n",
        "# as the row inside df2?\n",
        "\n",
        "# Let's take a look. Let's say our critical value is 0.1, or and alpha of 10%. And we're going to compare each\n",
        "# column in df1 to the same numbered column in df2. And we'll report when the p-value isn't less than 10%,\n",
        "# which means that we have sufficient evidence to say that the columns are different.\n",
        "\n",
        "# Let's write this in a function called test_columns\n",
        "def test_columns(alpha=0.1):\n",
        "    # I want to keep track of how many differ\n",
        "    num_diff=0\n",
        "    # And now we can just iterate over the columns\n",
        "    for col in df1.columns:\n",
        "        # we can run out ttest_ind between the two dataframes\n",
        "        teststat,pval=ttest_ind(df1[col],df2[col])\n",
        "        # and we check the pvalue versus the alpha\n",
        "        if pval<=alpha:\n",
        "            # And now we'll just print out if they are different and increment the num_diff\n",
        "            print(\"Col {} is statistically significantly different at alpha={}, pval={}\".format(col,alpha,pval))\n",
        "            num_diff=num_diff+1\n",
        "    # and let's print out some summary stats\n",
        "    print(\"Total number different was {}, which is {}%\".format(num_diff,float(num_diff)/len(df1.columns)*100))\n",
        "\n",
        "# And now lets actually run this\n",
        "test_columns()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Col 4 is statistically significantly different at alpha=0.1, pval=0.03171904508748956\n",
            "Col 9 is statistically significantly different at alpha=0.1, pval=0.05535264960034999\n",
            "Col 13 is statistically significantly different at alpha=0.1, pval=0.06411026546897207\n",
            "Col 25 is statistically significantly different at alpha=0.1, pval=0.06474837389971169\n",
            "Col 46 is statistically significantly different at alpha=0.1, pval=0.0766378775191977\n",
            "Col 73 is statistically significantly different at alpha=0.1, pval=0.021821747279110435\n",
            "Col 76 is statistically significantly different at alpha=0.1, pval=0.04067259983835964\n",
            "Col 85 is statistically significantly different at alpha=0.1, pval=0.0029530262143926777\n",
            "Col 86 is statistically significantly different at alpha=0.1, pval=0.010704932840894129\n",
            "Total number different was 9, which is 9.0%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dYZ5PgFQDLi1",
        "outputId": "bba8b673-5724-41b4-d3ea-818dd6ac69e3"
      },
      "source": [
        "# Interesting, so we see that there are a bunch of columns that are different! In fact, that number looks a\n",
        "# lot like the alpha value we chose. So what's going on - shouldn't all of the columns be the same? Remember\n",
        "# that all the ttest does is check if two sets are similar given some level of confidence, in our case, 10%.\n",
        "# The more random comparisons you do, the more will just happen to be the same by chance. In this example, we\n",
        "# checked 100 columns, so we would expect there to be roughly 10 of them if our alpha was 0.1.\n",
        "\n",
        "# We can test some other alpha values as well\n",
        "test_columns(0.05)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Col 4 is statistically significantly different at alpha=0.05, pval=0.03171904508748956\n",
            "Col 73 is statistically significantly different at alpha=0.05, pval=0.021821747279110435\n",
            "Col 76 is statistically significantly different at alpha=0.05, pval=0.04067259983835964\n",
            "Col 85 is statistically significantly different at alpha=0.05, pval=0.0029530262143926777\n",
            "Col 86 is statistically significantly different at alpha=0.05, pval=0.010704932840894129\n",
            "Total number different was 5, which is 5.0%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DBCI7QgJDLi2",
        "outputId": "d6044cfc-0610-4522-afd2-84d627224ad6"
      },
      "source": [
        "# So, keep this in mind when you are doing statistical tests like the t-test which has a p-value. Understand\n",
        "# that this p-value isn't magic, that it's a threshold for you when reporting results and trying to answer\n",
        "# your hypothesis. What's a reasonable threshold? Depends on your question, and you need to engage domain\n",
        "# experts to better understand what they would consider significant.\n",
        "\n",
        "# Just for fun, lets recreate that second dataframe using a non-normal distribution, I'll arbitrarily chose\n",
        "# chi squared\n",
        "df2=pd.DataFrame([np.random.chisquare(df=1,size=100) for x in range(100)])\n",
        "test_columns()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Col 0 is statistically significantly different at alpha=0.1, pval=0.0006359131132062404\n",
            "Col 1 is statistically significantly different at alpha=0.1, pval=1.6254596067473314e-05\n",
            "Col 2 is statistically significantly different at alpha=0.1, pval=0.032780760709076824\n",
            "Col 3 is statistically significantly different at alpha=0.1, pval=0.0032178974546995965\n",
            "Col 4 is statistically significantly different at alpha=0.1, pval=0.0433143837061582\n",
            "Col 5 is statistically significantly different at alpha=0.1, pval=7.931565914508593e-05\n",
            "Col 6 is statistically significantly different at alpha=0.1, pval=0.0017013488226864455\n",
            "Col 7 is statistically significantly different at alpha=0.1, pval=0.00035664966978204754\n",
            "Col 8 is statistically significantly different at alpha=0.1, pval=0.0008750262277182237\n",
            "Col 9 is statistically significantly different at alpha=0.1, pval=3.9005870973813224e-05\n",
            "Col 10 is statistically significantly different at alpha=0.1, pval=0.0001340016519264235\n",
            "Col 11 is statistically significantly different at alpha=0.1, pval=2.9387732963990273e-06\n",
            "Col 12 is statistically significantly different at alpha=0.1, pval=0.0002593817469470156\n",
            "Col 13 is statistically significantly different at alpha=0.1, pval=0.0001639697026470583\n",
            "Col 14 is statistically significantly different at alpha=0.1, pval=0.00015526373271627754\n",
            "Col 15 is statistically significantly different at alpha=0.1, pval=4.1837854872688554e-05\n",
            "Col 16 is statistically significantly different at alpha=0.1, pval=0.00139904683631816\n",
            "Col 17 is statistically significantly different at alpha=0.1, pval=0.008340935008253801\n",
            "Col 18 is statistically significantly different at alpha=0.1, pval=0.0005255784024108116\n",
            "Col 19 is statistically significantly different at alpha=0.1, pval=0.0010508383648962462\n",
            "Col 20 is statistically significantly different at alpha=0.1, pval=9.250384193688497e-05\n",
            "Col 21 is statistically significantly different at alpha=0.1, pval=0.0004139078346825869\n",
            "Col 22 is statistically significantly different at alpha=0.1, pval=0.000239886090894122\n",
            "Col 23 is statistically significantly different at alpha=0.1, pval=0.005114973931480402\n",
            "Col 24 is statistically significantly different at alpha=0.1, pval=3.9612287309026586e-06\n",
            "Col 25 is statistically significantly different at alpha=0.1, pval=0.001183662110645545\n",
            "Col 26 is statistically significantly different at alpha=0.1, pval=0.0013250467524673893\n",
            "Col 27 is statistically significantly different at alpha=0.1, pval=0.0071413662391291675\n",
            "Col 28 is statistically significantly different at alpha=0.1, pval=0.0018050525585905151\n",
            "Col 29 is statistically significantly different at alpha=0.1, pval=1.8443168650897388e-07\n",
            "Col 30 is statistically significantly different at alpha=0.1, pval=0.00018610780116814143\n",
            "Col 31 is statistically significantly different at alpha=0.1, pval=0.003288231435508567\n",
            "Col 32 is statistically significantly different at alpha=0.1, pval=0.00022788735453261414\n",
            "Col 33 is statistically significantly different at alpha=0.1, pval=0.0005420802253194416\n",
            "Col 34 is statistically significantly different at alpha=0.1, pval=0.00043337835781774326\n",
            "Col 35 is statistically significantly different at alpha=0.1, pval=0.0009296626942443332\n",
            "Col 36 is statistically significantly different at alpha=0.1, pval=0.015486530171592511\n",
            "Col 37 is statistically significantly different at alpha=0.1, pval=0.08968808146980235\n",
            "Col 38 is statistically significantly different at alpha=0.1, pval=0.0003984738782135047\n",
            "Col 39 is statistically significantly different at alpha=0.1, pval=0.00029812833313821313\n",
            "Col 40 is statistically significantly different at alpha=0.1, pval=0.00020580570179961344\n",
            "Col 41 is statistically significantly different at alpha=0.1, pval=0.016524830474479658\n",
            "Col 42 is statistically significantly different at alpha=0.1, pval=0.004414785565347055\n",
            "Col 43 is statistically significantly different at alpha=0.1, pval=0.0006514221896553882\n",
            "Col 44 is statistically significantly different at alpha=0.1, pval=0.0009246080966774699\n",
            "Col 45 is statistically significantly different at alpha=0.1, pval=3.947719385039122e-05\n",
            "Col 46 is statistically significantly different at alpha=0.1, pval=0.003936803652126271\n",
            "Col 47 is statistically significantly different at alpha=0.1, pval=0.00021471514569468722\n",
            "Col 48 is statistically significantly different at alpha=0.1, pval=0.00021533428873488688\n",
            "Col 49 is statistically significantly different at alpha=0.1, pval=0.08198381088074698\n",
            "Col 50 is statistically significantly different at alpha=0.1, pval=0.0005833899282268582\n",
            "Col 51 is statistically significantly different at alpha=0.1, pval=0.0008976227172661998\n",
            "Col 52 is statistically significantly different at alpha=0.1, pval=0.002183481627605226\n",
            "Col 53 is statistically significantly different at alpha=0.1, pval=0.00011517331466172489\n",
            "Col 54 is statistically significantly different at alpha=0.1, pval=0.0004539298416324909\n",
            "Col 55 is statistically significantly different at alpha=0.1, pval=0.0005972354479418133\n",
            "Col 56 is statistically significantly different at alpha=0.1, pval=0.00039537371324602574\n",
            "Col 57 is statistically significantly different at alpha=0.1, pval=1.5992684064779836e-05\n",
            "Col 58 is statistically significantly different at alpha=0.1, pval=0.0035581119921681737\n",
            "Col 59 is statistically significantly different at alpha=0.1, pval=0.0009069465410284678\n",
            "Col 60 is statistically significantly different at alpha=0.1, pval=0.001490617627902622\n",
            "Col 61 is statistically significantly different at alpha=0.1, pval=9.982665694645061e-05\n",
            "Col 62 is statistically significantly different at alpha=0.1, pval=0.0001855201502830867\n",
            "Col 63 is statistically significantly different at alpha=0.1, pval=0.0717892199552874\n",
            "Col 64 is statistically significantly different at alpha=0.1, pval=0.00011111316914952978\n",
            "Col 65 is statistically significantly different at alpha=0.1, pval=0.0007326247241706332\n",
            "Col 66 is statistically significantly different at alpha=0.1, pval=0.0006985304647035251\n",
            "Col 67 is statistically significantly different at alpha=0.1, pval=4.043055942025079e-05\n",
            "Col 68 is statistically significantly different at alpha=0.1, pval=0.0030620341209672865\n",
            "Col 69 is statistically significantly different at alpha=0.1, pval=0.0002566485047083988\n",
            "Col 70 is statistically significantly different at alpha=0.1, pval=1.5265122338601147e-05\n",
            "Col 71 is statistically significantly different at alpha=0.1, pval=0.0025159633542264875\n",
            "Col 72 is statistically significantly different at alpha=0.1, pval=0.006683482780335058\n",
            "Col 73 is statistically significantly different at alpha=0.1, pval=0.0004996694763506213\n",
            "Col 74 is statistically significantly different at alpha=0.1, pval=0.00024394144210986108\n",
            "Col 75 is statistically significantly different at alpha=0.1, pval=7.077435724052419e-05\n",
            "Col 76 is statistically significantly different at alpha=0.1, pval=0.0074093508129584405\n",
            "Col 77 is statistically significantly different at alpha=0.1, pval=0.006900999886679228\n",
            "Col 78 is statistically significantly different at alpha=0.1, pval=0.0010484064032203217\n",
            "Col 79 is statistically significantly different at alpha=0.1, pval=0.0023846343634806146\n",
            "Col 80 is statistically significantly different at alpha=0.1, pval=8.180792170754975e-05\n",
            "Col 81 is statistically significantly different at alpha=0.1, pval=5.49471394287242e-05\n",
            "Col 82 is statistically significantly different at alpha=0.1, pval=0.00033420295079797965\n",
            "Col 83 is statistically significantly different at alpha=0.1, pval=0.003652590090290096\n",
            "Col 84 is statistically significantly different at alpha=0.1, pval=0.0005087442921717129\n",
            "Col 85 is statistically significantly different at alpha=0.1, pval=0.010877718451872447\n",
            "Col 86 is statistically significantly different at alpha=0.1, pval=0.006487775025921745\n",
            "Col 87 is statistically significantly different at alpha=0.1, pval=0.00012983278069724697\n",
            "Col 88 is statistically significantly different at alpha=0.1, pval=0.0003217680289398193\n",
            "Col 89 is statistically significantly different at alpha=0.1, pval=0.0007595005579977705\n",
            "Col 90 is statistically significantly different at alpha=0.1, pval=0.0009007314132232909\n",
            "Col 91 is statistically significantly different at alpha=0.1, pval=0.0041773223441325755\n",
            "Col 92 is statistically significantly different at alpha=0.1, pval=0.00013412252503388242\n",
            "Col 93 is statistically significantly different at alpha=0.1, pval=0.0038256024065565237\n",
            "Col 94 is statistically significantly different at alpha=0.1, pval=0.0025884060593381543\n",
            "Col 95 is statistically significantly different at alpha=0.1, pval=0.0008645653623897333\n",
            "Col 96 is statistically significantly different at alpha=0.1, pval=0.00033563002443083584\n",
            "Col 97 is statistically significantly different at alpha=0.1, pval=9.000223928968477e-06\n",
            "Col 98 is statistically significantly different at alpha=0.1, pval=0.007777985999968882\n",
            "Col 99 is statistically significantly different at alpha=0.1, pval=0.00024422963411248024\n",
            "Total number different was 100, which is 100.0%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n85i8xLdDLi2"
      },
      "source": [
        "# Now we see that all or most columns test to be statistically significant at the 10% level."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "exqIbtZcDLi2"
      },
      "source": [
        "In this lecture, we've discussed just some of the basics of hypothesis testing in Python. I introduced you\n",
        "to the SciPy library, which you can use for the students t test. We've discussed some of the practical\n",
        "issues which arise from looking for statistical significance. There's much more to learn about hypothesis\n",
        "testing, for instance, there are different tests used, depending on the shape of your data and different\n",
        "ways to report results instead of just p-values such as confidence intervals or bayesian analyses. But this\n",
        "should give you a basic idea of where to start when comparing two populations for differences, which is a\n",
        "common task for data scientists."
      ]
    }
  ]
}