notebooks/week6_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 6  - An introduction to machine learning (Part II) - Exercise and Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll apply some of the material from the previous lectures to recreating the analysis from a [nature machine intelligence](https://www.nature.com/natmachintell/) paper, [\"An interpretable mortality prediction model for COVID-19 patients\"](https://www.nature.com/articles/s42256-020-0180-7)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Setup\n",
    "\n",
    "You will need to install the [xlrd] (https://xlrd.readthedocs.io/en/latest/) package to complete the Exercise.\n",
    "\n",
    " To install this packages, launch the \"Anaconda Prompt (Anaconda3)\" program and run:\n",
    "\n",
    "`conda install -c anaconda xlrd `\n",
    "\n",
    "<img src=\"../img/az_conda_prompt.png\">\n",
    "\n",
    "\n",
    "### Training data\n",
    "\n",
    "The original training datasets for the paper are linked as [Supplementary data](https://static-content.springer.com/esm/art%3A10.1038%2Fs42256-020-0180-7/MediaObjects/42256_2020_180_MOESM3_ESM.zip). You don't have to download this since we have included the single Excel file we need for this example as `data/time_series_375_preprocess_en.xlsx` in this project. Below we provide code to read the Excel data into a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_PATH = '../data/time_series_375_preprocess_en.xlsx'\n",
    "RANDOM_SEED=42"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_training_data(path):\n",
    "    \"\"\" Load Excel sheet of measurements from patients (timepandas.DataFrame with MultiIndex ['PATIENT_ID', 'RE_DATE'] (the unique patient identifier and patient sample date, corresponding to columns [0,1] respectively of the loaded worksheet), then retain the last set of measurements made per patient, drop 'Admission time', 'Discharge time', 'gender' and 'age' features, and replace NaNs with -1. \n",
    "    \"\"\"\n",
    "\n",
    "    # Specify explicitly what columns we want to load and what their data types are expected to be.\n",
    "    DTYPES = {\n",
    "        'PATIENT_ID': int,\n",
    "        'RE_DATE': str,\n",
    "        'age': int,\n",
    "        'gender': int,\n",
    "        'Admission time': str,\n",
    "        'Discharge time': str,\n",
    "        'outcome': float,\n",
    "        'Hypersensitive cardiac troponinI': float,\n",
    "        'hemoglobin': float,\n",
    "        'Serum chloride': float,\n",
    "        'Prothrombin time': float,\n",
    "        'procalcitonin': float,\n",
    "        'eosinophils(%)': float,\n",
    "        'Interleukin 2 receptor': float,\n",
    "        'Alkaline phosphatase': float,\n",
    "        'albumin': float,\n",
    "        'basophil(%)': float,\n",
    "        'Interleukin 10': float,\n",
    "        'Total bilirubin': float,\n",
    "        'Platelet count': float,\n",
    "        'monocytes(%)': float,\n",
    "        'antithrombin': float,\n",
    "        'Interleukin 8': float,\n",
    "        'indirect bilirubin': float,\n",
    "        'Red blood cell distribution width': float,\n",
    "        'neutrophils(%)': float,\n",
    "        'total protein': float,\n",
    "        'Quantification of Treponema pallidum antibodies': float,\n",
    "        'Prothrombin activity': float,\n",
    "        'HBsAg': float,\n",
    "        'mean corpuscular volume': float,\n",
    "        'hematocrit': float,\n",
    "        'White blood cell count': float,\n",
    "        'Tumor necrosis factorα': float,\n",
    "        'mean corpuscular hemoglobin concentration': float,\n",
    "        'fibrinogen': float,\n",
    "        'Interleukin 1β': float,\n",
    "        'Urea': float,\n",
    "        'lymphocyte count': float,\n",
    "        'PH value': float,\n",
    "        'Red blood cell count': float,\n",
    "        'Eosinophil count': float,\n",
    "        'Corrected calcium': float,\n",
    "        'Serum potassium': float,\n",
    "        'glucose': float,\n",
    "        'neutrophils count': float,\n",
    "        'Direct bilirubin': float,\n",
    "        'Mean platelet volume': float,\n",
    "        'ferritin': float,\n",
    "        'RBC distribution width SD': float,\n",
    "        'Thrombin time': float,\n",
    "        '(%)lymphocyte': float,\n",
    "        'HCV antibody quantification': float,\n",
    "        'D-D dimer': float,\n",
    "        'Total cholesterol': float,\n",
    "        'aspartate aminotransferase': float,\n",
    "        'Uric acid': float,\n",
    "        'HCO3-': float,\n",
    "        'calcium': float,\n",
    "        'Amino-terminal brain natriuretic peptide precursor(NT-proBNP)': float,\n",
    "        'Lactate dehydrogenase': float,\n",
    "        'platelet large cell ratio ': float,\n",
    "        'Interleukin 6': float,\n",
    "        'Fibrin degradation products': float,\n",
    "        'monocytes count': float,\n",
    "        'PLT distribution width': float,\n",
    "        'globulin': float,\n",
    "        'γ-glutamyl transpeptidase': float,\n",
    "        'International standard ratio': float,\n",
    "        'basophil count(#)': float,\n",
    "        '2019-nCoV nucleic acid detection': float,\n",
    "        'mean corpuscular hemoglobin': float,\n",
    "        'Activation of partial thromboplastin time': float,\n",
    "        'High sensitivity C-reactive protein': float,\n",
    "        'HIV antibody quantification': float,\n",
    "        'serum sodium': float,\n",
    "        'thrombocytocrit': float,\n",
    "        'ESR': float,\n",
    "        'glutamic-pyruvic transaminase': float,\n",
    "        'eGFR': float,\n",
    "        'creatinine': float\n",
    "    }\n",
    "\n",
    "    # Specify which string columns should be interpreted as datetimes.\n",
    "    DATETIME_COLUMNS = ['RE_DATE', 'Admission time', 'Discharge time']\n",
    "    \n",
    "    return (\n",
    "        pd.read_excel(path, index_col=[0,1], dtype=DTYPES, parse_dates=DATETIME_COLUMNS)\n",
    "            .sort_index()\n",
    "            .groupby('PATIENT_ID').last()\n",
    "            .drop(['Admission time', 'Discharge time'], axis=1)\n",
    "            .drop(['age', 'gender'], axis=1) # removed in later preprocessing step in original paper       \n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_columns_with_missing_data(df, threshold=0.2):\n",
    "    \"\"\" Remove all columns from DataFrame df where the proportion of missing records is greater than threshold.\n",
    "    \"\"\"\n",
    "    return df.dropna(axis=1, thresh=(1.0-threshold)*len(df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(375, 75)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>outcome</th>\n",
       "      <th>Hypersensitive cardiac troponinI</th>\n",
       "      <th>hemoglobin</th>\n",
       "      <th>Serum chloride</th>\n",
       "      <th>Prothrombin time</th>\n",
       "      <th>procalcitonin</th>\n",
       "      <th>eosinophils(%)</th>\n",
       "      <th>Interleukin 2 receptor</th>\n",
       "      <th>Alkaline phosphatase</th>\n",
       "      <th>albumin</th>\n",
       "      <th>...</th>\n",
       "      <th>mean corpuscular hemoglobin</th>\n",
       "      <th>Activation of partial thromboplastin time</th>\n",
       "      <th>High sensitivity C-reactive protein</th>\n",
       "      <th>HIV antibody quantification</th>\n",
       "      <th>serum sodium</th>\n",
       "      <th>thrombocytocrit</th>\n",
       "      <th>ESR</th>\n",
       "      <th>glutamic-pyruvic transaminase</th>\n",
       "      <th>eGFR</th>\n",
       "      <th>creatinine</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PATIENT_ID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>19.9</td>\n",
       "      <td>131.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>12.4</td>\n",
       "      <td>0.09</td>\n",
       "      <td>1.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>71.0</td>\n",
       "      <td>37.6</td>\n",
       "      <td>...</td>\n",
       "      <td>32.3</td>\n",
       "      <td>38.9</td>\n",
       "      <td>2.6</td>\n",
       "      <td>0.09</td>\n",
       "      <td>142.7</td>\n",
       "      <td>0.16</td>\n",
       "      <td>41.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>74.7</td>\n",
       "      <td>88.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>149.0</td>\n",
       "      <td>98.1</td>\n",
       "      <td>12.3</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.1</td>\n",
       "      <td>441.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>37.2</td>\n",
       "      <td>...</td>\n",
       "      <td>32.2</td>\n",
       "      <td>36.0</td>\n",
       "      <td>27.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>137.4</td>\n",
       "      <td>0.27</td>\n",
       "      <td>40.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>94.6</td>\n",
       "      <td>74.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>126.0</td>\n",
       "      <td>102.2</td>\n",
       "      <td>13.6</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.1</td>\n",
       "      <td>591.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>38.4</td>\n",
       "      <td>...</td>\n",
       "      <td>33.3</td>\n",
       "      <td>34.8</td>\n",
       "      <td>3.6</td>\n",
       "      <td>0.10</td>\n",
       "      <td>143.2</td>\n",
       "      <td>0.23</td>\n",
       "      <td>29.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>84.6</td>\n",
       "      <td>64.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>4.8</td>\n",
       "      <td>103.0</td>\n",
       "      <td>103.1</td>\n",
       "      <td>16.3</td>\n",
       "      <td>0.38</td>\n",
       "      <td>2.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>79.0</td>\n",
       "      <td>34.1</td>\n",
       "      <td>...</td>\n",
       "      <td>39.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.5</td>\n",
       "      <td>0.11</td>\n",
       "      <td>144.2</td>\n",
       "      <td>0.27</td>\n",
       "      <td>72.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>74.2</td>\n",
       "      <td>88.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.0</td>\n",
       "      <td>5.6</td>\n",
       "      <td>130.0</td>\n",
       "      <td>102.2</td>\n",
       "      <td>14.6</td>\n",
       "      <td>0.02</td>\n",
       "      <td>3.0</td>\n",
       "      <td>258.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>39.5</td>\n",
       "      <td>...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.08</td>\n",
       "      <td>143.6</td>\n",
       "      <td>0.36</td>\n",
       "      <td>11.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>122.8</td>\n",
       "      <td>54.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 75 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            outcome  Hypersensitive cardiac troponinI  hemoglobin  \\\n",
       "PATIENT_ID                                                          \n",
       "1               0.0                              19.9       131.0   \n",
       "2               0.0                               1.9       149.0   \n",
       "3               0.0                               NaN       126.0   \n",
       "4               0.0                               4.8       103.0   \n",
       "5               0.0                               5.6       130.0   \n",
       "\n",
       "            Serum chloride  Prothrombin time  procalcitonin  eosinophils(%)  \\\n",
       "PATIENT_ID                                                                    \n",
       "1                    100.0              12.4           0.09             1.7   \n",
       "2                     98.1              12.3           0.09             0.1   \n",
       "3                    102.2              13.6           0.06             0.1   \n",
       "4                    103.1              16.3           0.38             2.5   \n",
       "5                    102.2              14.6           0.02             3.0   \n",
       "\n",
       "            Interleukin 2 receptor  Alkaline phosphatase  albumin  ...  \\\n",
       "PATIENT_ID                                                         ...   \n",
       "1                              NaN                  71.0     37.6  ...   \n",
       "2                            441.0                  45.0     37.2  ...   \n",
       "3                            591.0                  69.0     38.4  ...   \n",
       "4                              NaN                  79.0     34.1  ...   \n",
       "5                            258.0                  84.0     39.5  ...   \n",
       "\n",
       "            mean corpuscular hemoglobin   \\\n",
       "PATIENT_ID                                 \n",
       "1                                   32.3   \n",
       "2                                   32.2   \n",
       "3                                   33.3   \n",
       "4                                   39.2   \n",
       "5                                   30.0   \n",
       "\n",
       "            Activation of partial thromboplastin time  \\\n",
       "PATIENT_ID                                              \n",
       "1                                                38.9   \n",
       "2                                                36.0   \n",
       "3                                                34.8   \n",
       "4                                                 NaN   \n",
       "5                                                 NaN   \n",
       "\n",
       "            High sensitivity C-reactive protein  HIV antibody quantification  \\\n",
       "PATIENT_ID                                                                     \n",
       "1                                           2.6                         0.09   \n",
       "2                                          27.4                          NaN   \n",
       "3                                           3.6                         0.10   \n",
       "4                                          14.5                         0.11   \n",
       "5                                           0.8                         0.08   \n",
       "\n",
       "            serum sodium  thrombocytocrit   ESR  \\\n",
       "PATIENT_ID                                        \n",
       "1                  142.7             0.16  41.0   \n",
       "2                  137.4             0.27  40.0   \n",
       "3                  143.2             0.23  29.0   \n",
       "4                  144.2             0.27  72.0   \n",
       "5                  143.6             0.36  11.0   \n",
       "\n",
       "            glutamic-pyruvic transaminase   eGFR  creatinine  \n",
       "PATIENT_ID                                                    \n",
       "1                                    30.0   74.7        88.0  \n",
       "2                                    22.0   94.6        74.0  \n",
       "3                                    67.0   84.6        64.0  \n",
       "4                                    26.0   74.2        88.0  \n",
       "5                                    18.0  122.8        54.0  \n",
       "\n",
       "[5 rows x 75 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = load_training_data(path=TRAIN_PATH)\n",
    "print(data.shape)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To set things up, as done in the paper, we'll remove all the columns with more than 20% missing data, and separate out our predictors ('X') and response ('y') variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = remove_columns_with_missing_data(data).fillna(-1)\n",
    "X = data.drop('outcome', axis=1)\n",
    "y = data.outcome.astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Split data into training and test sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Fit a RandomForestClassifier on the training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Evaluate the classifier performance by calculating the confusion matrix and the [F1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Plot the feature importances of the fitted classifier (this is basically the main finding of the Nature paper)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Try running a different type of classifier and/or see how well you can do on the test set by tuning hyperparameters using cross-validation, grid search or otherwise."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}