14. OGrisel Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictive Modeling with heterogeneous data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import warnings\n",
    "warnings.simplefilter('ignore', DeprecationWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading tabular data from the Titanic kaggle challenge in a pandas Data Frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us have a look at the Titanic dataset from the Kaggle Getting Started challenge at:\n",
    "\n",
    "https://www.kaggle.com/c/titanic-gettingStarted\n",
    "\n",
    "We can load the CSV file as a pandas data frame in one line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked\n",
      "\n",
      "1,0,3,\"Braund, Mr. Owen Harris\",male,22,1,0,A/5 21171,7.25,,S\n",
      "\n",
      "2,1,1,\"Cumings, Mrs. John Bradley (Florence Briggs Thayer)\",female,38,1,0,PC 17599,71.2833,C85,C\n",
      "\n",
      "3,1,3,\"Heikkinen, Miss. Laina\",female,26,0,0,STON/O2. 3101282,7.925,,S\n",
      "\n",
      "4,1,1,\"Futrelle, Mrs. Jacques Heath (Lily May Peel)\",female,35,1,0,113803,53.1,C123,S\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('data/titanic_train.csv', 'r') as f:\n",
    "    for i, line in zip(range(5), f):\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('data/titanic_train.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas data frames have a HTML table representation in the IPython notebook. Let's have a look at the first 5 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId    891\n",
       "Survived       891\n",
       "Pclass         891\n",
       "Name           891\n",
       "Sex            891\n",
       "Age            714\n",
       "SibSp          891\n",
       "Parch          891\n",
       "Ticket         891\n",
       "Fare           891\n",
       "Cabin          204\n",
       "Embarked       889\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data frame has 891 rows. Some passengers have missing information though: in particular Age and Cabin info can be missing. The meaning of the columns is explained on the challenge website:\n",
    "\n",
    "https://www.kaggle.com/c/titanic-gettingStarted/data\n",
    "\n",
    "A data frame can be converted into a numpy array by calling the `values` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['PassengerId',\n",
       " 'Survived',\n",
       " 'Pclass',\n",
       " 'Name',\n",
       " 'Sex',\n",
       " 'Age',\n",
       " 'SibSp',\n",
       " 'Parch',\n",
       " 'Ticket',\n",
       " 'Fare',\n",
       " 'Cabin',\n",
       " 'Embarked']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(891, 12)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 3, ..., 7.25, nan, 'S'],\n",
       "       [2, 1, 1, ..., 71.2833, 'C85', 'C'],\n",
       "       [3, 1, 3, ..., 7.925, nan, 'S'],\n",
       "       ...,\n",
       "       [889, 0, 3, ..., 23.45, nan, 'S'],\n",
       "       [890, 1, 1, ..., 30.0, 'C148', 'C'],\n",
       "       [891, 0, 3, ..., 7.75, nan, 'Q']], dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However this cannot be directly fed to a scikit-learn model:\n",
    "\n",
    "\n",
    "- the target variable (survival) is mixed with the input data\n",
    "\n",
    "- some attribute such as unique ids have no predictive values for the task\n",
    "\n",
    "- the values are heterogeneous (string labels for categories, integers and floating point numbers)\n",
    "\n",
    "- some attribute values are missing (nan: \"not a number\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting survival"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of the challenge is to predict whether a passenger has survived from others known attribute. Let us have a look at the `Survived` columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "survived_column = data['Survived']\n",
    "survived_column.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`data.Survived` is an instance of the pandas `Series` class with an integer dtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(survived_column)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `data` object is an instance pandas `DataFrame` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Series` can be seen as homegeneous, 1D columns. `DataFrame` instances are heterogenous collections of columns with the same length.\n",
    "\n",
    "The original data frame can be aggregated by counting rows for each possible value of the `Survived` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Survived</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>549</td>\n",
       "      <td>549</td>\n",
       "      <td>549</td>\n",
       "      <td>549</td>\n",
       "      <td>424</td>\n",
       "      <td>549</td>\n",
       "      <td>549</td>\n",
       "      <td>549</td>\n",
       "      <td>549</td>\n",
       "      <td>68</td>\n",
       "      <td>549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>342</td>\n",
       "      <td>342</td>\n",
       "      <td>342</td>\n",
       "      <td>342</td>\n",
       "      <td>290</td>\n",
       "      <td>342</td>\n",
       "      <td>342</td>\n",
       "      <td>342</td>\n",
       "      <td>342</td>\n",
       "      <td>136</td>\n",
       "      <td>340</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          PassengerId  Pclass  Name  Sex  Age  SibSp  Parch  Ticket  Fare  \\\n",
       "Survived                                                                    \n",
       "0                 549     549   549  549  424    549    549     549   549   \n",
       "1                 342     342   342  342  290    342    342     342   342   \n",
       "\n",
       "          Cabin  Embarked  \n",
       "Survived                   \n",
       "0            68       549  \n",
       "1           136       340  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.groupby('Survived').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6161616161616161"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(survived_column == 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this the subset of the full passengers list, about 2/3 perished in the event. So if we are to build a predictive model from this data, a baseline model to compare the performance to would be to always predict death. Such a constant model would reach around 62% predictive accuracy (which is higher than predicting at random):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas `Series` instances can be converted to regular 1D numpy arrays by using the `values` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = survived_column.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 1, 0], dtype=int64)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a predictive model on numerical features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sklearn` estimators all work with homegeneous numerical feature descriptors passed as a numpy array. Therefore passing the raw data frame will not work out of the box.\n",
    "\n",
    "Let us start simple and build a first model that only uses readily available numerical features as input, namely `data.Fare`, `data.Pclass` and `data.Age`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.2500</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>71.2833</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.9250</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53.1000</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.0500</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass   Age\n",
       "0   7.2500       3  22.0\n",
       "1  71.2833       1  38.0\n",
       "2   7.9250       3  26.0\n",
       "3  53.1000       1  35.0\n",
       "4   8.0500       3  35.0"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numerical_features = data.get(['Fare', 'Pclass', 'Age'])\n",
    "numerical_features.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately some passengers do not have age information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fare      891\n",
       "Pclass    891\n",
       "Age       714\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numerical_features.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use pandas `fillna` method to input the median age for those passengers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fare      15.7417\n",
       "Pclass     2.0000\n",
       "Age       28.0000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "median_features = numerical_features.dropna().median()\n",
    "median_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fare      891\n",
       "Pclass    891\n",
       "Age       891\n",
       "dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imputed_features = numerical_features.fillna(median_features)\n",
    "imputed_features.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.2500</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>71.2833</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.9250</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53.1000</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.0500</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass   Age\n",
       "0   7.2500       3  22.0\n",
       "1  71.2833       1  38.0\n",
       "2   7.9250       3  26.0\n",
       "3  53.1000       1  35.0\n",
       "4   8.0500       3  35.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imputed_features.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data frame is clean, we can convert it into an homogeneous numpy array of floating point values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 7.25  ,  3.    , 22.    ],\n",
       "       [71.2833,  1.    , 38.    ],\n",
       "       [ 7.925 ,  3.    , 26.    ],\n",
       "       ...,\n",
       "       [23.45  ,  3.    , 28.    ],\n",
       "       [30.    ,  1.    , 26.    ],\n",
       "       [ 7.75  ,  3.    , 32.    ]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_array = imputed_features.values\n",
    "features_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_array.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take the 80% of the data for training a first model and keep 20% for computing is generalization score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "features_train, features_test, target_train, target_test = train_test_split(\n",
    "    features_array, target, test_size=0.20, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(712, 3)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(179, 3)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(712,)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(179,)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with a simple model from sklearn, namely `LogisticRegression`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='liblinear', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "logreg = LogisticRegression(C=1, solver='liblinear')\n",
    "logreg.fit(features_train, target_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_predicted = logreg.predict(features_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7318435754189944"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "accuracy_score(target_test, target_predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This first model has around 73% accuracy: this is better than our baseline that always predicts death."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7318435754189944"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg.score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model evaluation and interpretation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting linear model weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `coef_` attribute of a fitted linear model such as `LogisticRegression` holds the weights of each features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Fare', 'Pclass', 'Age'], dtype='object')"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_names = numerical_features.columns\n",
    "feature_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.0043996 , -0.80916725, -0.03348064]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFH1JREFUeJzt3XuUHnd93/H3B+NLGgLI2LpwEeIiGnNxnLAYY47TUFvlWuRQQ+00RPREUe4JvaRV66RNaElEnDTQJifJIqesQyg0LiAVTIy8GJJw8fE6YINxyBoC2PFWkkWgOE4Mtb/945k1y/Ksnl3Ps1rZv/frHJ2Zeea38/3tGe18nvnNM8+kqpAktetha90BSdLaMggkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjXv4WndgKaeddlpt2bJlrbshSQ8qN9xww51VdfpKfua4DYItW7YwMzOz1t2QpAeVJF9Y6c84NCRJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaN5YgSPKiJJ9JcmuS3UPWn5zkHd3665JsGUddSVJ/vYMgyQnAbwEvBp4OXJLk6Yua/TDw11X1VOA3gDf0rStJGo9x3FB2NnBrVX0OIMnbge3Apxe02Q78Yjd/JfCbSVKr+MDkLbvfu1qbbt7n97x0rbsgaYzGEQSPA25bsHw78Nyl2lTV/0vyFeAxwJ0LGyXZBewC2Lx5c69OebB6cDG4V89q/S24z1bPsT5+jeMaQYa8tvid/nLaUFWTVTVRVROnn76ir8qQJD1A4wiC24EnLFh+PHDHUm2SPBx4FPClMdSWJPU0jiC4Htia5ElJTgIuBvYvarMf2NHNXwR8YDWvD0iSlq/3NYJuzP+ngKuBE4Dfq6qbk7wOmKmq/cDlwO8nuZXBmcDFfetKksZjLF9DXVVXAVcteu0/LJj/O+CV46glSRov7yyWpMYZBJLUuOP2CWWSjm/eq/PQ4RmBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhrXKwiSnJrkQJLZbrpuiXZ/lOTLSd7Tp54kafz6nhHsBqaraisw3S0Pcxnw6p61JEmroG8QbAemuvkp4MJhjapqGvhqz1qSpFXQNwg2VNUcQDdd32djSXYlmUkyc/jw4Z5dkyQtx8gH0yS5Btg4ZNWl4+5MVU0CkwATExM17u1Lkr7VyCCoqguWWpfkYJJNVTWXZBNwaKy9kyStur5DQ/uBHd38DmBfz+1Jko6xvkGwB9iWZBbY1i2TZCLJ3vlGSf4E+EPg/CS3J3lhz7qSpDHp9fD6qjoCnD/k9Rlg54Ll8/rUkSStHu8slqTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXG9giDJqUkOJJntpuuGtDkryUeT3JzkpiT/tE9NSdJ49T0j2A1MV9VWYLpbXuxu4Ieq6hnAi4A3Jnl0z7qSpDHpGwTbgalufgq4cHGDqvqLqprt5u9g8ID703vWlSSNSd8g2FBVcwDddP3RGic5GzgJ+GzPupKkMRn5zOIk1wAbh6y6dCWFkmwCfh/YUVX3LdFmF7ALYPPmzSvZvCTpARoZBFV1wVLrkhxMsqmq5roD/aEl2j0SeC/w81X1saPUmgQmASYmJmpU3yRJ/fUdGtoP7OjmdwD7FjdIchLwLuCKqvrDnvUkSWPWNwj2ANuSzALbumWSTCTZ27V5FfC9wGuSfKL7d1bPupKkMRk5NHQ0VXUEOH/I6zPAzm7+rcBb+9SRJK0e7yyWpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDWuVxAkOTXJgSSz3XTdkDZPTHJD92Sym5P8WJ+akqTx6ntGsBuYrqqtwHS3vNgccG5VnQU8F9id5LE960qSxqRvEGwHprr5KeDCxQ2q6mtVdU+3ePIYakqSxqjvQXlDVc0BdNP1wxoleUKSm4DbgDdU1R1LtNuVZCbJzOHDh3t2TZK0HCMfXp/kGmDjkFWXLrdIVd0GnNkNCb07yZVVdXBIu0lgEmBiYqKWu31J0gM3Mgiq6oKl1iU5mGRTVc0l2QQcGrGtO5LcDJwHXLni3kqSxq7v0NB+YEc3vwPYt7hBkscn+bZufh3wfOAzPetKksakbxDsAbYlmQW2dcskmUiyt2tzBnBdkhuBDwG/VlWf7FlXkjQmI4eGjqaqjgDnD3l9BtjZzR8AzuxTR5K0evwopyQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY3rFQRJTk1yIMlsN113lLaPTPJXSX6zT01J0nj1PSPYDUxX1VZgulteyn9i8IQySdJxpG8QbAemuvkp4MJhjZI8G9gAvL9nPUnSmPUNgg1VNQfQTdcvbpDkYcCvAz83amNJdiWZSTJz+PDhnl2TJC3HyGcWJ7kG2Dhk1aXLrPETwFVVdVuSozasqklgEmBiYqKWuX1JUg8jg6CqLlhqXZKDSTZV1VySTcChIc2eB5yX5CeARwAnJbmrqo52PUGSdIyMDIIR9gM7gD3ddN/iBlX1z+bnk7wGmDAEJOn40fcawR5gW5JZYFu3TJKJJHv7dk6StPp6nRFU1RHg/CGvzwA7h7z+FuAtfWpKksbLO4slqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqXK8gSHJqkgNJZrvpuiXa3ZvkE92//X1qSpLGq+8ZwW5guqq2AtPd8jB/W1Vndf9e3rOmJGmM+gbBdmCqm58CLuy5PUnSMdY3CDZU1RxAN12/RLtTkswk+VgSw0KSjiMjn1mc5Bpg45BVl66gzuaquiPJk4EPJPlkVX12SK1dwC6AzZs3r2DzkqQHamQQVNUFS61LcjDJpqqaS7IJOLTENu7opp9L8kHgu4FvCYKqmgQmASYmJmpZv4EkqZe+Q0P7gR3d/A5g3+IGSdYlObmbPw14PvDpnnUlSWPSNwj2ANuSzALbumWSTCTZ27U5A5hJciNwLbCnqgwCSTpOjBwaOpqqOgKcP+T1GWBnN/8R4Fl96kiSVo93FktS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktS4XkGQ5NQkB5LMdtN1S7TbnOT9SW5J8ukkW/rUlSSNT98zgt3AdFVtBaa75WGuAC6rqjOAs1ni2caSpGOvbxBsB6a6+SngwsUNkjwdeHhVHQCoqruq6u6edSVJY9I3CDZU1RxAN10/pM3TgC8neWeSjye5LMkJPetKksZk5DOLk1wDbByy6tIV1DgP+G7gi8A7gNcAlw+ptQvYBbB58+Zlbl6S1MfIIKiqC5Zal+Rgkk1VNZdkE8PH/m8HPl5Vn+t+5t3AOQwJgqqaBCYBJiYmanm/giSpj75DQ/uBHd38DmDfkDbXA+uSnN4t/0Pg0z3rSpLGpG8Q7AG2JZkFtnXLJJlIshegqu4F/jUwneSTQIA396wrSRqTkUNDR1NVR4Dzh7w+A+xcsHwAOLNPLUnS6vDOYklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS43oFQZJTkxxIMttN1w1p84Ikn1jw7++SXNinriRpfPqeEewGpqtqKzDdLX+Tqrq2qs6qqrMYPK/4buD9PetKksakbxBsB6a6+Slg1Dv9i4D3VdXdPetKksakbxBsqKo5gG66fkT7i4H/sdTKJLuSzCSZOXz4cM+uSZKWY+TD65NcA2wcsurSlRRKsgl4FnD1Um2qahKYBJiYmKiVbF+S9MCMDIKqumCpdUkOJtlUVXPdgf7QUTb1KuBdVfX1B9BPSdIq6Ts0tB/Y0c3vAPYdpe0lHGVYSJK0NvoGwR5gW5JZYFu3TJKJJHvnGyXZAjwB+FDPepKkMRs5NHQ0VXUEOH/I6zPAzgXLnwce16eWJGl1eGexJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjesVBElOTXIgyWw3XbdEu19NcnOSW5L81yTpU1eSND59zwh2A9NVtRWY7pa/SZJzgecDZwLPBJ4D/IOedSVJY9I3CLYDU938FHDhkDYFnAKcBJwMnAgc7FlXkjQmfYNgQ1XNAXTT9YsbVNVHgWuBue7f1VV1S8+6kqQxGfnM4iTXABuHrLp0OQWSPBU4A3h899KBJN9bVX88pO0uYBfA5s2bl7N5SVJPI4Ogqi5Yal2Sg0k2VdVckk3AoSHNvh/4WFXd1f3M+4BzgG8JgqqaBCYBJiYmanm/giSpj5FBMMJ+YAewp5vuG9Lmi8CPJPkVIAwuFL+xZ109xHx+z0vXugtSs/peI9gDbEsyC2zrlkkykWRv1+ZK4LPAJ4EbgRur6n/3rCtJGpNeZwRVdQQ4f8jrM8DObv5e4Ef71JEkrR7vLJakxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXGpOj6/ySHJYeALa90P9XIacOdad0Ir4j578Fm8z55YVaevZAPHbRDowS/JTFVNrHU/tHzuswefcewzh4YkqXEGgSQ1ziDQappc6w5oxdxnDz6995nXCCSpcZ4RSFLjDAJJapxBoGMqSda6D9JDUZITuumK/8YMAh0TSR4GUAsuShkKDz7zBxsdP+b/jrqHgAGcsnjdKAaBjomqug8gycuTXJbkLOCRa9wtrdD8wSbJJUkesdb90TfeXCV5RZI/Af5jktcuXDeKQaBVM38W0M2fmORNwI8BtwC/AOxYq75peeb34fw7yyQXJfkY8I9Z8M5Tx9bCv61u+Wzgh4BdwA3Aa5Ocu9ztGQRaNVV1X5LvSPLSqvr64KV6CfAo4Azgz9e2hzqaJCfMn8kB80MMLwF+sap+oKruTNLruedamfmhue5v68QkZ3ah8GzgA8D3Af8e+OWq+shyt+tO1KpJ8lLg1cChJNcBz03yl8C7gXOr6stJNgIHl3sKq2Onqu5N8hhgD3Bzkt8D7gX+eZIXAo8H5pJcXlU3rmVfW7FgaO7VwCsYHMN/ELgVuAr4b8A5VXVPktOB9VV186jtekag3jKw+FT1DOB1wKGq+hng68BHgXdU1b/oQuAiYCfw7ce80/oWQ4aBnsNgn90GvK2q/i+DA800cCXw34GTgCesSYcbkOQFSZ60YPnRSf4AeCHwDuBZwCuBWeBtDN5U3ZPkAgb76Izl1PGMQL117+YryZOBrcCfVdUtSa4EXt61+UqS9wE/muStwMkMDiCvq6q71qrv+qZPdN3XTefPzs4B3gi8BdiY5GlV9afATd3PnQt8D4MDjsYsyakMDu6zSaaq6vJu1Qbg4m5o7iTg+QyuC7wemOquF2wC/ktVLWvf+BUTWrFunPJ1wFVV9eHuHeQe4EUM3qX8EwbvWGBwIPlwVf1297OnA2cDm6pq7zHvvO6X5O9V1d0Llp8B/CxwLYNhhmcy2H9HGFzP+UHgTcAVwL8BJoBfqKr3H+OuNyHJoxgEwXsYDLH+DnAd8JPAvqqa7tpdz2Cf7QYKeEpV3bpgOxk19OrQkFYkyU7gGuApwKe6l58C3MXgAH898AzgTVV1J/BO4MVJHgtQVYer6r2GwNpK8mJgd5JN3fJPA28FPsLgwuPrGbzLfBFwSVW9lsGY9N+vqi8Ak1X1XENgdXQH768AXwIewSCgnwdczODTWk9N8sSu+Q3A4xgEQM2HwIILyyPf7RsEWrYk6xl80+FPV9XF3XDPd3T/8X4D+FXgXwLnAtuTnFtV7wT+GnjZmnVc91t0Q9g6BgcXGFxsfB7wVwyG854M/ExVHQFOTvKTDK4PXA/gxeFj5l3AiVV1PfAZBh+5/k4Gw6q/leSDwN8AT2Twd3e/BTeYjeQ1Ai1bVR3qPjlyBvCpJFcAB4GfA04EHg38SFXdnuQm4A+AJwH/qjs70Brphu8yf3Coqvcl2Qqck+Tj3fIrgB9n8K7zTOAVSf4Xg+s+3we8uqpuWpvfoC0L3sV/O3BWkrcz2Ce/BFzI4BrAdQyGZ29IMgX85QOt5xmBVupngbd1B/qbGYxLwuAdyinAC5L8W+AA8LsAhsDamh8j7j57/qQkv5PkeQyuAzwc2NY13Qa8var+jMEB6DHA9qr6o6p6pSGwJt7DYL8crqqnV9UU8O8YXMD/z8DZST7O4NNCf/xAi3hGoBWpqr9Jsgt4WVW9Ae4/0NyU5H8C388gEH68qg6vZV9b1g0B/QDwp8AdwD1JtgOXAXuBT1XVV5N8CviuJJuBDwGvT/JdwFnAz1fVtWvzG6jzZQYH/avg/pv8/pzuZswktzMI6y/2KeKnhrRi3ccNvwi8pAuAk6rqa926k6vqnrXtYduS/DCD+zO+yuAi/oer6te7r/j4YFW9a0HbxwI/BXypqn6tu1Hs2cCbDfK11w3p7QPeDLxnwfcKjfwk0Ep4RqAV64YYXsngwvE58yHQrTME1lB3Qf/NwHdW1V90++ll3Q1+9wBP69qdWFVfr6o7uu8OelWS51TV1cDVa/YL6JtUVSV5TVV9afHr46zjNQI9IFX1UeC+JGeudV/0DVV1CLicb4z7zwDrgb8FPgt8W5JnVtXXkzw5ySVVtR/4le6TKTrOzIfA/B3fq8EgUB/neQHxuPRa4JeSnMLgW0LvA/4P8EEG3xV0RZJfZjDkcDrAcr6PRmtrNb+Py2sE0kNQd+Pf7wK/DVzW3QQ2v+4fMfgs+rv7XmTUQ4NBID0EdcMItwPfU1UHk5wMfM1vedUwDg1JD0HdAf8iYH+3fI8hoKUYBNJDVHdB/14v6GsUh4akh7DuBqRlf+eM2mQQSFLjHBqSpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjfv/+UQgBx1oKvYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(len(feature_names))\n",
    "plt.bar(x, logreg.coef_.ravel())\n",
    "_ = plt.xticks(x + 0.5, feature_names, rotation=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, survival is slightly positively linked with Fare (the higher the fare, the higher the likelyhood the model will predict survival) while passenger from first class and lower ages are predicted to survive more often than older people from the 3rd class.\n",
    "\n",
    "First-class cabins were closer to the lifeboats and children and women reportedly had the priority. Our model seems to capture that historical data. We will see later if the sex of the passenger can be used as an informative predictor to increase the predictive accuracy of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alternative evaluation metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to see the details of the false positive and false negative errors by computing the confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[98 12]\n",
      " [36 33]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "cm = confusion_matrix(target_test, target_predicted)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The true labeling are seen as the rows and the predicted labels are the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATsAAAEYCAYAAAAj5FFfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcXEW5//HPdxKWhB0CGEAIYAAFBRLgh3BZBGRRBFxAFlG4EQS9LCIKbldAuIKgiKKyKpuyiOT+ENSAKLLIlmDYZJMlsgQIyE6AJDz3j6qGZpheZqZ7Tk+f75vXeU33OafrPD0kT+pUnapSRGBm1u16ig7AzGwoONmZWSk42ZlZKTjZmVkpONmZWSk42ZlZKTjZWUtJGiXpd5Kel/SbQZSzh6QrWhlbUSRtIuneouMoO/k5u3KStDtwCLAG8CIwHTgmIq4bZLl7AgcAG0XE3EEH2uEkBTA+Iv5ZdCxWn2t2JSTpEOBHwP8AywIrAj8DdmxB8SsB95Uh0TVD0siiY7AsIryVaAMWA14Cdq5zzgKkZPh43n4ELJCPbQ48CnwFeAqYCeydjx0JvA7MydeYBBwBnFdV9jgggJH5/V7Ag6Ta5UPAHlX7r6v63EbALcDz+edGVceuBr4LXJ/LuQIYU+O7VeL/WlX8OwEfAe4D/g18o+r8DYAbgOfyuScD8+dj1+Tv8nL+vp+uKv8w4Ang3Mq+/JlV8zUm5PfLAU8Dmxf9Z6Pbt8ID8DbE/8NhW2BuJdnUOOco4EZgGWBp4G/Ad/OxzfPnjwLmy0niFWCJfLx3cquZ7ICFgBeA1fOxscCa+fWbyQ5YEngW2DN/brf8fql8/GrgAWA1YFR+f2yN71aJ/79z/PsAs4BfA4sAawKvAqvk8ycCG+brjgPuBg6uKi+A9/RR/nGkfzRGVSe7fM4+uZzRwBTghKL/XJRh821s+SwFPB31bzP3AI6KiKciYhapxrZn1fE5+ficiPg9qVaz+gDjeQNYS9KoiJgZEXf1cc5Hgfsj4tyImBsR5wP3AB+rOueXEXFfRMwGLgLWqXPNOaT2yTnABcAY4KSIeDFf/y7gAwARMS0ibszXfRg4Fdisie/0nYh4LcfzNhFxOnA/cBMpwX+zQXnWAk525fMMMKZBW9JywIyq9zPyvjfL6JUsXwEW7m8gEfEy6dZvP2CmpMslrdFEPJWYlq96/0Q/4nkmIubl15Vk9GTV8dmVz0taTdJlkp6Q9AKpnXNMnbIBZkXEqw3OOR1YC/hJRLzW4FxrASe78rmBdJu2U51zHid1NFSsmPcNxMuk27WKd1UfjIgpEfFhUg3nHlISaBRPJabHBhhTf/ycFNf4iFgU+AagBp+p+4iDpIVJ7aBnAkdIWrIVgVp9TnYlExHPk9qrfippJ0mjJc0naTtJ38+nnQ98S9LSksbk888b4CWnA5tKWlHSYsDXKwckLStpB0kLAa+Rbofn9VHG74HVJO0uaaSkTwPvAy4bYEz9sQipXfGlXOvcv9fxJ4FV+lnmScC0iPg8cDlwyqCjtIac7EooIn5IesbuW6TG+UeA/wL+N59yNDAVuB24A7g17xvIta4ELsxlTePtCaqH1Kv7OKmHcjPgi32U8QywfT73GVJP6vYR8fRAYuqnQ4HdSb28p5O+S7UjgLMlPSdpl0aFSdqR1Em0X951CDBB0h4ti9j65IeKzawUXLMzs1JwsjOzUnCyM7NScLIzs1LwIOUhopGjQvMvUnQYXWXd965YdAhd6dZbpz0dEUu3oqwRi64UMfcdg0jeFLNnTYmIbVtxrUac7IaI5l+EBVZv+GSC9cP1N51cdAhdadR86j1aZcBi7uy6f+5fnf7TRqNRWsbJzszaR4KeEUVHATjZmVm7qTO6BpzszKyNXLMzs7JQo3kThoaTnZm1j9vszKw03GZnZt3PNTszKwPhNjszKwNBT2ekmc6Iwsy6V49rdmbW7UTHtNl1RjeJmXUppd7YWlszJUgHSbpT0l2SDs77lpR0paT7888lGpXjZGdm7dUzovbWgKS1SIuKbwCsDWwvaTxwOHBVRIwHrsrv64cxqC9hZlaPVH9r7L3AjRHxSl6r+K/Ax4EdgbPzOWdTf2lQwMnOzNqtfs1ujKSpVdu+vT59J2kpzqUkjQY+ArwbWDYiZgLkn8s0CsMdFGbWRmrUNvd0RKxX62BE3C3pOOBK0rrCtwFzBxKJa3Zm1j6V3tgBttkBRMSZETEhIjYlrS98P/CkpLEA+edTjcpxsjOzNmpJb+wy+eeKwCeA84FLgc/lUz4H/P9G5fg21szaa/DP2f1W0lLAHOBLEfGspGOBiyRNAv4F7NyoECc7M2uvQY6NjYhN+tj3DLBlf8pxsjOz9vF8dmZWFvKsJ2bW7SSQJwIws+4n1+zMrBx6ejrjCTcnOzNrK9fszKzrSXKbnZmVg2t2ZlYKbrMzs+6nvHUAJzszaxuhjqnZdUYUZta1JNXcmvz8l/P6E3dKOl/SgpJWlnRTXoPiQknzNyrHyc7M2iePoKi1Nfy4tDxwILBeRKwFjAB2BY4DTsxrUDwLTGpUlpOdmbXVYGt2pOa2UZJGAqOBmcAWwMX5eFNrULjNzszapok2uzGSpla9Py0iTqu8iYjHJJ1AmrNuNnAFMA14Li/AA/AosHyjWJzszKy96lfg6q5BkdeD3RFYGXgO+A2wXR+nRqMwnOzMrH006OfstgIeiohZAJIuATYCFpc0MtfuVgAeb1SQ2+zMrK0G2Wb3L2BDSaOVPrAl8A/gL8Cn8jlNrUHRMclO0l6Slhvia/6tReWcJelTjc80K5/B9MZGxE2kjohbgTtIOes04DDgEEn/BJYCzmxUVifdxu5FWhC3YXW0WflfAkXEG30dj4iNWnUtM3unfva69ikivgN8p9fuB4EN+lNOW2p2ksZJulvS6flhwCskjcrH1pF0o6TbJU2WtESuFa0H/ErS9Mq5VeUdKOkf+TMX5H1HSDq06pw783Ur1/4Z6V+Db0v6ftV5e0n6SX79Uv55oaSPVJ1zlqRPShoh6XhJt+RrfyEfl6STc0yX08Rq5GZl1dPTU3Mb0jjaWPZ44KcRsSapF+WTef85wGER8QFStfQ7EXExMBXYIyLWiYjZvco6HFg3f2a/Jq69OnBORKwL/Iy01mTFp4ELe51/Qd5PfhJ7S+D3pAcVn4+I9YH1gX0krQx8PF/j/cA+pAbTd5C0r6SpkqbG3N5fyawkVGcbQu1Mdg9FxPT8ehowTtJiwOIR8de8/2xg0ybKup1U6/sMMLfRycCMiLgRIPfiPChpw7z25OrA9b3O/wOwhaQFSN3a1+SEuzXwWUnTgZtIbQPjc8znR8S8iHgc+HNfQUTEaRGxXkSsp5Gj+jrFrLupc2p27Wyze63q9TxgMH/bP0pKMDuQbkvXJCW96t/WglWvX+71+QuBXYB7gMkR8bZnciLiVUlXA9uQanjn50MCDoiIKdXn51vehs/1mJWdGPSysS0zpKk1Ip4HnpVUWfR2T6BSy3sRWKT3ZyT1AO+OiL8AXwMWBxYGHgYm5HMmkB46rOUS0nCS3XjnLWzFBcDewCZAJblNAfaXNF++zmqSFgKuAXbNbXpjgQ/V/+ZmZSV6empvQ6mI3tjPAadIGk3qUdk77z8r758NfLCq3W4EcF6+BRZp8O9zkn7LW7eYtwD31bpgRDwr6R/A+yLi5hqnXUFqT7w0Il7P+84AxgG35p7dWaSkOZk0Nu+OfN2/vqM0MwM6Z6Zi9bqjszbpGb1MLLD6LkWH0VWeveXkokPoSqPm07R6Q7j6VdbY1WLlvWv/f7r7e9u07FqNdNJzdmbWhTqkYudkZ2ZtJIa8ba4WJzsza5vUG+tkZ2Zdb+h7XWvpmIkAzKw7DWbWE0mr5yGkle0FSQdLWlLSlUprUFyZ572ry8nOzNpGuc1uoM/ZRcS9eQjpOsBE4BXSo1+HA1flNSiuyu/rcrIzs7aSam/9tCXwQETMIM1efHbe7zUozKx4DWpwddeg6GVX3hrKuWxEzASIiJmSGs485GRnZu2jhr2xddegeLOYNBvRDsDXBxqKk52ZtY1a1xu7HXBrRDyZ3z8paWyu1Y0FnmpUgNvszKytWtRmtxtv3cICXEoaZw9NrkHhmp2ZtU8LRlDkSUM+DHyhavexwEWSJpEW5dm5UTlOdmbWNq0YQRERr5Amzq3e9wypd7ZpNZOdpEUbBPBCfy5kZuXUKSMo6tXs7iLNxlsdaeV9ACu2MS4z6wbDYSKAiHj3UAZiZt1HDH4pxVZpqjdW0q6SvpFfryBpYnvDMrNuMaJHNbeh1DDZSTqZtMbCnnnXK8Ap7QzKzLpHC4eLDUozvbEbRcQESX8HiIh/56eZzczqkhjyGlwtzSS7OXmFrwDIa6++0daozKxrDKc2u58CvwWWlnQkcB1wXFujMrOuIKBHqrkNpYY1u4g4R9I0YKu8a+eIuLO9YZlZt+iQu9imR1CMAOaQbmU9ntbMmqNhNC27pG+SBuAuB6wA/FrSgKdZMbPyGFa3scBngIl5fBqSjgGmAd9rZ2Bm1h1aMBHA4sAZwFqku8v/BO4FLgTGAQ8Du0TEs3XjaOJaM3h7UhwJPNjviM2sdOo9Y9ePit1JwB8jYg1gbeBuBrAGRb2JAE4kZdFXgLskTcnvtyb1yJqZNTRiELereUKSTYG9ACLideB1STsCm+fTzgauBg6rV1a929hKj+tdwOVV+2/sb8BmVl4NnrNrtAbFKsAs4JeS1iY1oR1EK9egiIgzG33YzKweqeEY2EZrUIwEJgAHRMRNkk6iiVvWvjTTG7uqpAsk3S7pvso2kIuZWfkMss3uUeDRiLgpv7+YlPyezGtP0Mo1KM4CfknqRd4OuAi4oKkwzazUxOBmPYmIJ4BHJK2ed20J/IM2rUExOiKmSDohIh4AviXp2iY+Z2bWirGxBwC/yhOQPAjsTaqotXwNiteUon1A0n7AY0DDxkAzM2lwvbEAETEd6KtdrzVrUFT5MrAwcCBwDLAY6aE+M7OGOmTSk6YmAqg0DL7IWxN4mpk1pVPGxtZ7qHgyeQ67vkTEJ9oSkZl1DTH0Y2BrqVezO3nIoiiB8assx8/OP6LoMLrK9IefKzoEa2SYrC521VAGYmbdqVPmhGt2Pjszs36rPGfXCZzszKytOiTXNZ/sJC0QEa+1Mxgz6y6dtLpYM2NjN5B0B3B/fr+2pJ+0PTIz6wqdsm5sM22HPwa2B54BiIjbSItmm5nVJWCkVHMbSs3cxvZExIxe49vmtSkeM+syg81pkh4mDWqYB8yNiPUkLUkbpmV/RNIGQEgaIelgwFM8mVlDlfnsBjrrSZUPRcQ6VXPf9Xta9maS3f7AIcCKwJPAhnmfmVlDPaq9DcKOpOnYyT93avSBZsbGPgXsOqiwzKyUWvScXQBXSArg1Dxte+umZX8zWOl0+hgjGxH79j9mMyuVxjW4RmtQAGwcEY/nhHalpHsGEkozHRR/qnq9IPBx4JGBXMzMykU0nM+u0RoURMTj+edTeYKSDcjTsudaXVPTsjdzG3vh24KXzgWubPQ5MzMYXNucpIVIT4S8mF9vDRzFW9OyH0sLp2XvbWVgpQF8zsxKpgVtdssCk/OjbyOBX0fEHyXdQqunZZf0LG+12fUA/2aAS5mZWckMcqRERDwIrN3H/mdo5bTsee2JtUnrTgC8ERE1J/Q0M6smYORwGBubE9vkiJiXNyc6M+uX4TQ29mZJE9oeiZl1HSFGqPY2lOqtQTEyIuYC/wHsI+kB4GVSzTQiwgnQzOob/EiJlqnXZnczMIEmhmGYmfVluMxULICIeGCIYjGzLjQcVhdbWtIhtQ5GxA/bEI+ZdZE0gqLoKJJ6yW4EsDC5hmdm1m9K0zx1gnrJbmZEHDVkkZhZ12libOyQadhmZ2Y2GJ2SSOolu34NxTAzeyfR0yG9sTUfKo6Ifw9lIGbWfURKMrW2pstJS0L8XdJl+f3Kkm6SdL+kCyXN36iM/lzPzKzfeqSaWz8cBNxd9f444MS8BsWzwKSGcfQrajOz/si9sbW2poqQVgA+CpyR3wvYArg4n9KaNSjMzAaqid7YZqZl/xHwNWCR/H4p4Lk8nBXgUWD5RrE42ZlZWzWov9Wdll3S9sBTETFN0uZ1imw4I5OTnZm1TQues9sY2EHSR0hr4CxKquktXjVZyQrA440KcpudmbXVYOazi4ivR8QKETGOtKTrnyNiD+AvwKfyaU2tQeFkZ2ZtVLsndpATBBwGHCLpn6Q2vDMbfcC3sWbWNuk5u9Y8VBwRVwNX59cPkpZUbJqTnZm1j6CnQ+4fnezMrK3UIaNjnezMrG2Gy6wnZmaD1iG5rry9sZJ2kNSSxb4lvdSKcsy6TaVm19Gri3WDqocO3yEiLgUuHeKQzEpGHdNmNyxqdpIWknS5pNsk3Snp05IeljQmH19P0tX59RGSTpN0BXBOngZmzaqyrpY0UdJekk6WtFguqycfHy3pEUnzSVpV0h8lTZN0raQ18jkrS7pB0i2Svjv0vxGzYSIvpVhrG0rDItkB2wKPR8TaEbEW8McG508EdoyI3YELgF0AJI0FlouIaZUTI+J54DZgs7zrY8CUiJgDnAYcEBETgUOBn+VzTgJ+HhHrA0/UCkLSvpKmSpr63LPP9O8bm3UB0bIpngZtuCS7O4CtJB0naZOcoOq5NCJm59cXATvn17sAv+nj/AuBT+fXuwIXSloY2Aj4jaTpwKnA2HzOxsD5+fW5tYKIiNMiYr2IWG/xJZZqELJZdxrMcLFWGhZtdhFxn6SJwEeA7+Vb1Lm8lawX7PWRl6s++5ikZyR9gJTQvtDHJS7N5S5JqhX+GViINI3MOrXCGvAXMisRt9n1g6TlgFci4jzgBGAC8DApMQF8skERF5Dmw1osIu7ofTAiXgJuJt2eXhYR8yLiBeAhSTvnGCRp7fyR60k1QIA9BvzFzErAbXb9837g5nw7+U3gaOBI4CRJ1wLzGnz+YlJyuqjOORcCn8k/K/YAJkm6DbgL2DHvPwj4kqRbgMX6+V3MykV1tkYflRaUdHPunLxL0pF5f7/XoBgut7FTgCl9HFqtj3OP6GPfk/T6rhFxFnBW1fuL6fXrj4iHSJ0jvct7CPhg1a5j64RvVloSg+2IeA3YIiJekjQfcJ2kPwCHkNaguEDSKaQ1KH5er6DhUrMzs2FqEBU7Iqk8tD9f3oIBrEHhZGdmbVR7sZ284M6YyuNZedv3HSWkZRSnA08BVwIP4DUozKzTNLiLrbsGBUBEzAPWkbQ4MBl4b1+nNYrDyc7M2ka07nm6iHguj5TaEK9BYWadRnX+a/hZaelco0PSKGAr0mLZ/V6DwjU7M2urQT5PNxY4W9IIUuXsooi4TNI/gAskHQ38Ha9BYWaFarbbtYaIuB1Yt4/9XoPCzDpHZSKATuBkZ2Zt1RmpzsnOzNpMrtmZWRl0SK5zsjOz9nKyM7OulzpjOyPbOdmZWfsUMG9dLU52ZtZeTnZm1v2GfmGdWpzszKxtBjmAoqWc7MysrTrlOTvPemJmbTWYpRQlvVvSXyTdndegOCjvX1LSlXkNiislLdGoLCc7M2ufOiuLNdlLOxf4SkS8lzSP3ZckvQ84HLgqIsYDV+X3dTnZmVmbDXwVioiYGRG35tcvkuayW5600t/Z+bSm1qBwm52ZtU2a9aTuKWMkTa16f1pEnNZnWdI40nRPNwHLRsRMSAlR0jKNYnGyM7O2GuwaFKkMLQz8Fjg4Il4YSKeHb2PNrK0arC7WzOfnIyW6X0XEJXn3k5LG5uNjSSuP1eVkZ2ZtNZh1Y5Uy4pnA3RHxw6pDl5LWngCvQWFmRZMGPVPxxsCewB157ViAbwDHAhdJmgT8C9i5UUFOdmbWXoNbg+K6OiVs2Z+ynOzMrK0864mZlUBz68MOBSc7M2sb4ZmKzawknOzMrPsNvje2ZZzszKxtPJ+dmZVGp8xn52RnZm3VIbnOyc7M2svJzsxKoVOes1NEFB1DKUiaBcwoOo4mjQGeLjqILjOcfqcrRcTSrShI0h9J372WpyNi21Zcq2EsTnbWm6SpzcwxZs3z77R4nuLJzErByc7MSsHJzvrS5xoANij+nRbMbXZmVgqu2ZlZKTjZmVkpONnZgKlTBj0Oc/49Dg0nOxsQScsDO0laruhYhqtKkouIkDSi6Hi6nZOdDdSHgL2B3SVtLGlU0QENQz+QdImk+SNiHoAk/51sE/9ibUAi4jzg+6QVnn4B7Cdp7bygsTXnf0jTvc2QtA9ARLyhrNjQuo8fPbF+kzQyIuZKOhF4BngVWJP0j+cVwM0RcX+RMXa6XJt7XdL7ge+QasqzgAMj4op8jsJ/QVvGyc4GRNKKwOSImJjfLwQcA+wKfDkizi8yvuFA0pLA1cDuwCPAJ4DjgZuBSRExs7jouo9vY22gngHul/Q1SctExMsRcTBwI3BNwbENF+8H/hURdwIvRcQvga8A7wE2LDSyLuRkZ02rbkeKiJeBM4EVSL2y+0v6BfBARDxWVIzDzI1ASPpypYMCmAP8ICImFxhXV/JtrDWluv1I0qeB5YHpwHLAKsBSwCvAkRHxamGBDiOSFiX93iYDc4E/kG5pd4mIaUXG1o2c7KwpknpyT+EJpBmuVwVGR8SWkkZHxCsFh9jxJI2IiHmSdgA2AzYBLoiIH0raldTR82hETC000C7ladmtKTnRjQHWj4jNJJ0O3JIP7yDp7oi4rcAQO17VrephwIHAYsC7874pEfFsIYGVhNvsrD/eAP4s6Uhg5YioTFt0OOkvrjUgaTtSW93DwLrAf+dD35M0oai4ysDJzuqqPNEvaZm860lgZ+BcSStJ+ibwz4hwD2wNvUZFXEn6h+FS4KiIeFHSh4H1IuLWQgIsCd/GWl0R8UZ+uT/wQkScKGk0sA2wE2kRmYOLim84qPwOJe0HnA1cC3wc2EbS/MChpAeLrY3cQWFNkfRB4FTg3Ig4XtJKwPPAyxExp9joOlvVIzsXkEdJkHqzjwBmAtP8qEn7OdlZnyq9r/m18swc40gPvZ4XETcVGd9wJGlxUoK7NyJ+XnA4peM2O+tTVaLbHvi8pM8AjwFPAedI2qbI+IaTfNtPRDxHWotikqRjJC1QbGTl4mRn7yBpkqR1c8P6qsCSwOdJjeqzSY9L7FFgiB0vP6ZTcY2kX0j6GjAR+BKpJ3b9QoIrKXdQWF8eAx4i3bJeEhEPAMdJ2gAYAXwW+GeB8Q0He0q6GHgR+CjpIeIFSR0TiwJrAR8ArisswpJxm529Ta+2uqOBA4BLgP09DKw5eU6/VUn/IJwK3AH8OiKeysfHAvNFxL+Ki7J8fBtrb1OV6L4aEd8izcCxFPBIvg2zBiJiTkTcA4wFppB6Xo+W9HlJi0XETCe6oedkZ+8gaWlgnKS9I2JWROwA7Ah8WdIXCw5vWMiTch4SERcBJwDXA+8DTpI0sdDgSsq3sdYnSVsAPybNxHFURLxYcEjDSp6Y81zg36SmgNmkzokNgLM9DnboOdnZ20gaX5lSPa8gtg/wp4hwQ/oA5Flibo+Ic/L7URExu+CwSsm9sVY9fdNywMGSNgZ+TZqNeCLwMUlf9IPEtVU9eD2S1NO6MDAV+BVwnqRlgROd6IrjNjurHv/6fVLP64HA4sBo0p+R5UkTc1pjhwMnktbi+DuwKWmdiW8CKxUXlrlmV3JVE0quD8yOiKvyocosJj+RtHJEPFRQiB2vqla3KrA6sCfwBKkn+6Okef8WzM8rWkHcZmcASLoCWADYMiLm5n3zR8TrxUY2fEg6FFggIo4pOhZ7J9/GllivhZhPABYCbszjYXGia56kVUgrgn1S0ta57a5yzH/POoBrdiUnaTFgw4iYkt/vDUwiDXP6L996NU/SOqS2ugWB+4Br8jKJ1gH8L46tD/xC0hmS3pfXLt0J+Adp5TBrQNJOkn5HWgbx66QOifcAB0pas8jY7C2u2ZVQ9bKI+f3CwFHA1sAfgWMj4umi4htu8kSmewBbktaXOA6YD9guIs4rMjZ7i5NdieXFX+6NiAfz+4mkaZxmAFtHxEtFxtfJ+vgHY0FgNdL09WsAv4qIM4qKz97Jt7ElJWk8sBvwWUk7SFoyL8x8EnCOE119+VGTNSR9JT+U/Srp1v8nwHOkR1Csg7hmVyJ91EY2ALYClgFeIq1K/zFgo4h4rZgohw9JuwNHktaVOCYiLs8L6PwKODAiZhYaoL2NHyouFwGRlz+cGxHHSbqdNGvuRsAKpL+0TnQ1VD2EvQewZkSMl7QbcKKkHUnTOs1yous8TnYlkWt1b0hamdSY/uF86LOkdV9/UFx0w0dEzMsvDwT2za+fBW4gzft3DGmYmHUYJ7uSqLp93Ya0dumsPBPxRsB7JB2a516zBvKyks8DS0r6LrA2cDkwJiJuLDQ4q8kdFOVzB3AQ8DfgiYjYAjgeGF9oVMNIRNwA/J70e3s1T276CPCRQgOzutxBUUKSVgBWi4g/S1oduAjYyYP9m5eXQZw/Il7Mw8FuBr4dEX8oODSrwbexJVA1K8cIgIh4FHg0H94TuNSJrn9yJ85reQzsBsBkJ7rO5ppdF6vqOewBloqIWUXH1I3yhAo9VZ0X1oHcZtfdJua1EH4OHNb7YPXMHDZwkTjRdTj/Ye9S+eHWNYFTgGVJ417frO0BVOatMysD1+y6VES8nmcw+R3wIPDt/PDrgpLml/RDSaOKjdJs6LjNrgtVFtDJr5eJiKckbQV8lTTIfxFgdETsWGScZkPJt7FdpjJSIr8+nlSTWxH4JbAdqfd1DnBZcVGaDT0nu+5TGf/6VdIEkj8mrRL2ZWDtiDiyuuZnVhZOdl0mj38dAfwHcHRE3AIgaQbwNUnviognCg3SrADuoOhCubf1BtKwsMq+O0k1vVWLisusSO6g6BK9b00lLQr8FFgPOJe0ctjEiNi2oBDNCuVk12Uk7Q+8ThqYfhPpWbv9gD8Df4mIGQWGZ1YYJ7suUKnVSfoU8F3gL6Qe11eB/wVu9hP+VnZOdl1E0onAKRFxr6T/AP4faS2Eh4Dj3ANrZebe2C4haXvgAOBJ0lKI1+Up17cGHnWis7Jzza5L5LGwe5GmC78POCIibi80KLNW9S3kAAADVUlEQVQO4mTXZfIsJ18EdgamktZJeCP8P9pKzsmuS0laE9gqIk4qOhazTuBkZ2al4BEUZlYKTnZmVgpOdmZWCk52ZlYKTnbWNEnzJE2XdKek30gaPYiyNpd0WX69g6TD65y7uKQvDuAaR0g6tNn9vc45Kw+/a/Za4yTd2d8Ybeg42Vl/zI6IdSJiLdJkA/tVH1TS7z9TEXFpRBxb55TFSc8Omg2Yk50N1LXAe3KN5m5JPwNuBd4taWtJN0i6NdcAFwaQtK2keyRdB3yiUpCkvSSdnF8vK2mypNvythFwLLBqrlUen8/7qqRbJN0u6ciqsr4p6V5JfyKNC65L0j65nNsk/bZXbXUrSddKui8Px0PSCEnHV137C4P9RdrQcLKzfsvrzW4H3JF3rQ6cExHrAi8D3yI90DyBNIrjEEkLAqcDHwM2Ad5Vo/gfA3+NiLWBCcBdwOHAA7lW+VVJWwPjgQ2AdUjr424qaSKwK7AuKZmu38TXuSQi1s/XuxuYVHVsHLAZ8FHglPwdJgHPR8T6ufx9JK3cxHWsYJ4IwPpjlKTp+fW1wJnAcsCMiLgx798QeB9wvSSA+UmzJq8BPBQR9wNIOo80lK23LYDPwpszLj8vaYle52ydt7/n9wuTkt8iwOSIeCVf49ImvtNako4m3SovDEypOnZRnkDhfkkP5u+wNfCBqva8xfK172viWlYgJzvrj9kRsU71jpzQXq7eBVwZEbv1Om8doFXDdQR8LyJO7XWNgwdwjbOAnSLiNkl7AZtXHetdVuRrHxAR1UkRSeP6eV0bYr6NtVa7EdhY0nsAJI2WtBpwD7CypMoaGLvV+PxVwP75syPy9PIvkmptFVOA/6xqC1xe0jLANcDHJY2StAjplrmRRYCZkuYD9uh1bGdJPTnmVYB787X3z+cjaTVJCzVxHSuYa3bWUhExK9eQzpe0QN79rYi4T9K+wOWSngauA9bqo4iDgNMkTQLmAftHxA2Srs+Pdvwht9u9F7gh1yxfAj4TEbdKuhCYTloM/NomQv42afr6GaQ2yOqkei/wV2BZYL+IeFXSGaS2vFuVLj4L2Km5344VyRMBmFkp+DbWzErByc7MSsHJzsxKwcnOzErByc7MSsHJzsxKwcnOzErh/wA4Y6gM7umwzwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_confusion(cm):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=plt.cm.binary)\n",
    "    plt.title('Confusion matrix')\n",
    "    plt.set_cmap('Blues')\n",
    "    plt.colorbar()\n",
    "\n",
    "    target_names = ['not survived', 'survived']\n",
    "\n",
    "    tick_marks = np.arange(len(target_names))\n",
    "    plt.xticks(tick_marks, target_names, rotation=60)\n",
    "    plt.yticks(tick_marks, target_names)\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    # Convenience function to adjust plot parameters for a clear layout.\n",
    "    plt.tight_layout()\n",
    "    \n",
    "plot_confusion(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[98 12]\n",
      " [36 33]]\n"
     ]
    }
   ],
   "source": [
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can normalize the number of prediction by dividing by the total number of true \"survived\" and \"not survived\" to compute false and true positive rates for survival (in the second column of the confusion matrix)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.89090909 0.17391304]\n",
      " [0.32727273 0.47826087]]\n"
     ]
    }
   ],
   "source": [
    "print(cm.astype(np.float64) / cm.sum(axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can therefore observe that the fact that the target classes are not balanced in the dataset makes the accuracy score not very informative.\n",
    "\n",
    "scikit-learn provides alternative classification metrics to evaluate models performance on imbalanced data such as precision, recall and f1 score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "not survived       0.73      0.89      0.80       110\n",
      "    survived       0.73      0.48      0.58        69\n",
      "\n",
      "    accuracy                           0.73       179\n",
      "   macro avg       0.73      0.68      0.69       179\n",
      "weighted avg       0.73      0.73      0.72       179\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(target_test, target_predicted,\n",
    "                            target_names=['not survived', 'survived']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to quantify the quality of a binary classifier on imbalanced data is to compute the precision, recall and f1-score of a model (at the default fixed decision threshold of 0.5)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic Regression is a probabilistic models: instead of just predicting a binary outcome (survived or not) given the input features it can also estimates the posterior probability of the outcome given the input features using the `predict_proba` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.75263264, 0.24736736],\n",
       "       [0.75824771, 0.24175229],\n",
       "       [0.58542437, 0.41457563],\n",
       "       [0.25224882, 0.74775118],\n",
       "       [0.75817844, 0.24182156]])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_predicted_proba = logreg.predict_proba(features_test)\n",
    "target_predicted_proba[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default the decision threshold is 0.5: if we vary the decision threshold from 0 to 1 we could generate a family of binary classifier models that address all the possible trade offs between false positive and false negative prediction errors.\n",
    "\n",
    "We can summarize the performance of a binary classifier for all the possible thresholds by plotting the ROC curve and quantifying the Area under the ROC curve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import auc\n",
    "\n",
    "def plot_roc_curve(target_test, target_predicted_proba):\n",
    "    fpr, tpr, thresholds = roc_curve(target_test, target_predicted_proba[:, 1])\n",
    "    \n",
    "    roc_auc = auc(fpr, tpr)\n",
    "    # Plot ROC curve\n",
    "    plt.plot(fpr, tpr, label='ROC curve (area = %0.3f)' % roc_auc)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')  # random predictions curve\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.0])\n",
    "    plt.xlabel('False Positive Rate or (1 - Specifity)')\n",
    "    plt.ylabel('True Positive Rate or (Sensitivity)')\n",
    "    plt.title('Receiver Operating Characteristic')\n",
    "    plt.legend(loc=\"lower right\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4VGX2wPHvSQKE3kREQu+hQ+gCIlIEFKQXIXTrggpiQdcVu8sqq6KAAvKzYUVBUVi6KAKhNymCkKAgYOgEUs7vjzuJQ0gmQ8hkUs7neeZhbp0zl8mcue9773lFVTHGGGNSE+DvAIwxxmRtliiMMcZ4ZInCGGOMR5YojDHGeGSJwhhjjEeWKIwxxnhkicJ4TUQGichif8eRlYjIWRGp7IfXrSgiKiJBmf3aviAiO0Tk5nRsZ5/JTGCJIpsSkd9E5ILri+qIiLwnIoV8+Zqq+qGqdvTla7gTkZYiskxEzojIKRFZICKhmfX6KcSzQkRGus9T1UKqut9Hr1ddRD4TkeOu979VRB4WkUBfvF56uRJW1WvZh6rWVtUVabzOFckxsz+TuZUliuztdlUtBDQAGgKP+zmedEnpV7GItAAWA18DNwKVgC3Aj774BZ/VfpmLSBVgLRAJ1FXVokAfIAwonMGv5bf3ntWOu0mFqtojGz6A34Bb3aZfAb51m84HTAYOAUeBaUB+t+Xdgc3AaeBXoLNrflFgJvAHcBh4Dgh0LRsKrHY9nwZMThbT18DDruc3Al8Ax4ADwBi39f4FfA584Hr9kSm8vx+At1KY/x3wf67nNwNRwBPAcdcxGeTNMXDb9lHgCPA+UBz4xhVztOt5iGv954F4IAY4C7zpmq9AVdfz94CpwLfAGZwv+ipu8XQEdgOngLeAlSm9d9e6H7j/f6awvKLrtcNd7+84MNFteVNgDXDS9X/5JpDXbbkC9wN7gQOuef/FSUyngQ1Aa7f1A13H+VfXe9sAlANWufZ1znVc+rnW74bz+ToJ/ATUS/bZfRTYClwEgnD7PLtij3DFcRR41TX/kOu1zroeLXD7TLrWqQ38D/jLte0T/v5bzQkPvwdgj3T+x13+hxUCbAP+67Z8CjAfKIHzC3QB8KJrWVPXl1UHnLPKskBN17KvgOlAQeB6YB1wt2tZ0h8l0Mb1pSKu6eLABZwEEeD6IvknkBeoDOwHOrnW/RcQC/RwrZs/2XsrgPOl3C6F9z0M+MP1/GYgDngVJym0dX1h1fDiGCRu+7Jr2/xASaCX6/ULA58BX7m99gqSfbFzZaL4y3V8g4APgbmuZde5vvh6upaNdR2D1BLFEWCYh///iq7XfscVe32cL91aruWNgeau16oI7AIeTBb3/1zHJjF53uU6BkHAOFcMwa5lj+B8xmoA4nq9ksmPgWu6EfAn0AwnwYTjfF7zuX12N+Mkmvxu8xI/z2uAwa7nhYDmyd5zkNtrDeXvz2RhnKQ4Dgh2TTfz999qTnj4PQB7pPM/zvnDOovz606BpUAx1zLB+cJ0/zXbgr9/OU4HXkthn6VdXzbuZx4DgOWu5+5/lILzC6+Na3oUsMz1vBlwKNm+Hwdmu57/C1jl4b2FuN5TzRSWdQZiXc9vxvmyL+i2/FPgKS+Owc3ApcQvwlTiaABEu02vIO1E8a7bsi7AL67nQ4A1bssEJ9GmlihicZ3lpbI88UszxG3eOqB/Kus/CMxLFvctaXzGooH6rue7ge6prJc8UbwNPJtsnd1AW7fP7vAUPs+JiWIV8AxwXSrvObVEMQDY5Mu/u9z6sPbB7K2Hqi4RkbbARzi/Wk8CpXB+FW8QkcR1BefXHTi/5BamsL8KQB7gD7ftAnC+0C6jqioic3H+OFcBA3GaSxL3c6OInHTbJBCnOSnRFft0Ew0kAGWAX5ItK4PTzJK0rqqec5s+iHNWk9YxADimqjFJC0UKAK/hJKPirtmFRSRQVeM9xOvuiNvz8zi/iHHFlPSeXccvysN+TuC813S9nohUxznTCsM5DkE4Z3nuLvs/EJFxwEhXrAoUwflMgfOZ+dWLeMD5/w8XkX+4zcvr2m+Kr53MCGAS8IuIHACeUdVvvHjdq4nRXAXrzM4BVHUlzq/Zya5Zx3GagWqrajHXo6g6Hd/g/JFWSWFXkThnFNe5bVdEVWun8tIfA71FpALOWcQXbvs54LaPYqpaWFW7uIft4f2cw2l+6JPC4r44Z0+JiotIQbfp8sDvXhyDlGIYh9O00kxVi+A0r4GTYDzG7IU/cM6UnB062Ssk9dVZgtMMll5v4yTZaq738gR/v49ESe9HRFrj9Bv0BYqrajGc5snEbVL7zKQkEng+2f9/AVX9OKXXTk5V96rqAJymz5eBz13/x2kd/6uJ0VwFSxQ5xxSgg4g0UNUEnLbr10TkegARKSsinVzrzgSGiUh7EQlwLaupqn/gXGn0HxEp4lpWxXXGcgVV3YTT8fsusEhVE88g1gGnReRREckvIoEiUkdEmlzF+3kM51fpGBEpLCLFReQ5nOajZ5Kt+4yI5HV92XUDPvPiGKSkME5yOSkiJYCnky0/itPfkh7fAnVFpIfrSp/7gRs8rP800FJE/i0iN7jiryoiH4hIMS9erzBOn8hZEakJ3OvF+nE4/59BIvJPnDOKRO8Cz4pINXHUE5GSrmXJj8s7wD0i0sy1bkER6SoiXl2tJSJ3iUgp1/9h4mcq3hVbAqn/H3wD3CAiD4pIPtfnppk3r2k8s0SRQ6jqMeD/cNrnwfl1uA/4WURO4/xCreFadx1Op/BrOL8aV+I0F4DTlp4X2InTBPQ5nptAPgZuxWn6SowlHrgdp43/AM6v+3dxrqjy9v2sBjrhdP7+gdOk1BC4SVX3uq16xBXn7zidx/eoamJzVarHIBVTcDqGjwM/A98nW/5fnDOoaBF53dv34no/x3HOkF7BaVYKxbmy52Iq6/+KkxQrAjtE5BTOGVsETr9UWsbjNAeewfni/iSN9RfhXFG2B+dYx3B589CrOP0/i3ES0EycYwVOn9McETkpIn1VNQKnz+pNnP+bfTh9Cd7qjPOez+Ic8/6qGqOq53GuPvvR9VrN3TdS1TM4F2jcjvO52Au0u4rXNalIvGLFmGzHdSfvB6rqqQknSxKRAJzLcwep6nJ/x2OMJ3ZGYUwmEZFOIlJMRPLxd5/Bz34Oy5g0+SxRiMgsEflTRLanslxE5HUR2ecqTdDIV7EYk0W0wLkq5zhO80gPVb3g35CMSZvPmp5EpA3Odf7/p6p1UljeBfgHzrXmzXBuFrOOJ2OMyWJ8dkahqqtw7lJNTXecJKKq+jNQTES8uW7cGGNMJvLnDXdlufyqiijXvD+Srygio4HRAAULFmxcs2bNTAnQGGOyu4MHD3Lir2g0If64qpZKzz78mSiS3/wDqdxQo6ozgBkAYWFhGhER4cu4jDEmW0vsUhAR3n77bV78Yi2RS+ccTO/+/JkoonBuuU8UgnMtvDHG5Fq7j5zh1IXYdG//55HfeeWpR7i1aw869+hD4879uP5ENSKXzkn3Pv2ZKOYDD7jqBTUDTrnuDDbGmFwp8q/zdJqyKl3bqipntywievksSIhnR1AVZh3NmFuMfJYoRORjnAqd17mKnz2NU3AOVZ2GU5SuC85dm+dx7hQ2xphc6+zFOADGtq9G00olvN7u8MEDvPzkwxz6eTWNmt/Eo8+9StkKlS5b56aX0x+XzxKFq6iXp+WJA6cYY0yOdepCLLHxCV6tG33+EgC1yhSmVdXr0lj7b19tX82+nVuZMWMGI0eOxK1icoawMuPGGOMjP+07zsB31171dnkC075zYfv27WzcuJEhQ4bQo0cP9u/fT8mSJdPcLj0sURhjjI/8ecap+Ti2fTWuK5TXq22C8wR6PJu4dOkSL7zwAi+88AKlS5emb9++BAcH+yxJgCUKY4zxuR4Ny1LpuoJpr5iGtWvXMmLECHbs2MFdd93Fa6+9RnBwcAZE6JklCmOM8dL8Lb/z1FfbSfCy9FFi30RG9BgcPnyY1q1bU7p0ab755hu6du2aAXv1jiUKY4zx0s7fT3MmJpbwlhW93qZEgbyUL1Eg3a+5Z88eqlevTtmyZfnkk09o3749RYoUSXvDDGSJwhhjrkJQYABP357a6MAZ5+TJk0yYMIF3332XFStW0KZNG+68806fv25KLFEYY0wWM3/+fO69916OHDnCI488QpMmVzOKcMazRGGMMVnIyJEjmTlzJnXr1uXrr78mLCzM3yFZojDGGH9zL+IXFhZGhQoVePTRR8mb17tLan3NEoUxJlf5cmMUq/ceT9e2238/lcHRQGRkJPfccw/9+/dn8ODB3HPPPRn+GtfKEoUxJteY89NvPD1/B6UK5yM4T/rGbWtXI11DOlwhISGB6dOn8+ijjxIfH++3jmpvWKIwxuQKn6w/xNPzd3BrrdK8fVcjr8pk+MrevXsZOXIkq1at4tZbb2XGjBlUqlQp7Q39xBKFMSbHWLXnGFujTl4x/+T5WGb+eIC21UsxdVBDvyYJgJ07d7J161ZmzZrF0KFDM7yIX0azRGGMyTEmfrWNyL8upLjs5hqlmHZXY/IFBWZyVI4tW7awefNmwsPD6d69O/v376d48eJ+ieVqWaIwxuQYCQnQs1FZXu5V74pl/jqLuHjxIs899xwvvfQSZcqUoV+/fgQHB2ebJAGWKIzJFf44dYGNB69skslpzl+KI0DE701LidasWcOIESPYtWsXQ4YM4dVXX82UIn4ZzRKFMbnApAU7+W77EX+HkSmK5c/j7xAAp4hf27ZtueGGG1i4cCG33Xabv0NKN0sUxuQCMbHxVC9diDcHNvJ3KD6XEeW8r8WuXbuoVasWZcuW5dNPP6V9+/YULlzYrzFdK0sUxuQSwXkCqV46e39hZWXR0dGMGzeO2bNns2rVKlq3bk2PHj38HVaGsERhjDHXaN68edx3330cO3aMxx9/3O9F/DKaJQpjjLkGw4cPZ/bs2TRo0IBvv/2WRo1yXvOeV4lCRIoAZYALQKSql8M7GWMylapyyTWqmrt4+4vNUO5F/Jo3b061atUYP348efJkjY70jJZqohCRwsC9wECgEHAcCAZKishq4C1V/SFTojTGeGXM3M0s2PJ7issali+WydHkTAcPHuTuu+9m4MCBDBkyhNGjR/s7JJ/zdEYxD/gQaK+qJxJninOveVNgsIhUU9VZPo7RGOOl346fo3KpgvRqFHLFsuaVS/ghopwjISGBt99+m8ceewxVpU+fPv4OKdOkmihU9dZU5iuw1vUwxmQxFUsW5P52Vf0dRo6ye/duRo4cyerVq+nYsSPTp0+nYsWK/g4r06TZRyEic4HZwGLrmzAm63n66+0s2fUnAH+eiaFU4Ywpg23+tnv3bnbs2MF7773HkCFDsnwRv4zmTWf2e8Bw4E0R+QR4T1X3+TQqY4zXVu9zBuFpUaUkALfXv9Gf4eQYmzZtYvPmzQwbNow77riD/fv3U6xY7uznSTNRqOr3wPciUhwYBCwXkQPAO8DHqhrn4xiNMWloUL4Yk/vU93cYOUJMTAyTJk3ilVdeoWzZsgwYMIDg4OBcmyQAvKqc5UoSA4HBwFZgOtAS+N53oRljTOb68ccfadCgAS+++CJDhgxh8+bN2bKIX0bzpo/iU6Au8BHQS1WjXIs+FJFNvgzOGGMyy+HDh2nXrh1ly5Zl0aJFdOzY0d8hZRne9FG8C/zPvSNbRIJUNU5VG/ouNGOM8b2dO3cSGhpK2bJl+eKLL2jXrh2FChXyd1hZijeJ4iVVXZxs3jog592nbkwW8+Hag+w5csbjOsfOXKRmmUwKKAf566+/ePjhh5kzZw4rV66kTZs23H777f4OK0vydGf29ThlO/KLSF0g8XqwIkCBTIjNmFxv0oKdKFAgb+rDdwYECA1Ccm9Ha3p88cUX3H///Zw4cYKJEyfStGlTf4eUpXk6o+iKc1lsCPCW2/wzwFO+DMoY41BgeKtKPHZbTX+HkmMMHTqUOXPm0KhRI77//nsaNGjg75CyPE93Zs8GZotIX1X9NBNjMiZHUlUWbP2D6HOXvN4mPsHucc0I7kX8WrZsSa1atRg3bhxBQVZA2xuemp4GqOrHQBkRGZN8uaq+ntbORaQz8F8gEHhXVV9Ktrw8MAco5lrnMVVdeHVvwZjsISr6AmM+vvoLBW8sZpdnXosDBw4wevRo7rrrLsLDw3NFEb+M5imdFnf9e116diwigcBUoAMQBawXkfmqutNttSeBT1X1bREJBRYCFdPzesZkdXGus4PnetShS13vep8DBIoVyOvLsHKs+Ph4pk6dyuOPP05AQACDBg3yd0jZlqemp8R+iddU9a907LspsE9V90NSzajugHuiUJzOcYCiQMr1kY3JRlSVTZEnORtzedGCI6djACiUL4gSBe3L35d27drFiBEjWLNmDbfddhvTpk2jfPny/g4r2/KmgW69iOwGPgHmqeppL/ddFoh0m44CmiVb51/AYhH5B1AQSLFirYiMBkYD9p9tsrydf5ym51s/pbq8UD5rF/e1ffv2sXv3bt5//30GDRqU64r4ZTRvaj1VEZGWQH/gGRHZDMxV1blpbJrS/0zynrkBOEUG/yMiLYD3RaSOql42RJeqzgBmAISFhVnvnsnSLlyKB+CpbqE0KFf0smX5ggIJLVMkpc3MNdqwYQNbtmxh+PDh3H777Rw4cIAiRexYZwSvftqo6k/ATyLyL2AKzoBGaSWKKKCc23QIVzYtjQA6u15jjYgE4/SJ/OlNXMb427EzF7kYF3/FPIDqpQvRuIINFuRrFy5c4JlnnmHy5MmUK1eOgQMHEhwcbEkiA3lT66kQTt9Cf6AW8DVOQcC0rAeqiUgl4LBr+4HJ1jkEtAfeE5FaOEOtHvM6emP8aP1vf9Fn2ppUl+cN9KrmprkGq1atYuTIkezdu5cRI0YwefJkK+LnA96cUWwHFgCvXM0Y2aoaJyIPAItwLn2dpao7RGQSEKGq84FxwDsi8hBOs9RQGxzJZBcnzjpnDg/dWp0yyS5hLZg3iMYViqe0mckghw8fpn379pQrV44lS5bQvn17f4eUY3mTKCon7zPwluueiIXJ5v3T7flOoFV69m1MVtGxdmlqWb9Dptm2bRt169albNmyzJs3j3bt2lGwYEF/h5Wjebrh7j+qOg74QkSu+JWvqj19GpkxfrTsl6M88NGmpHsfUpLgWhZgV9RkiuPHj/PQQw/xwQcfJBXx69atm7/DyhU8nVF84vr3zcwIxJis5Nc/z3H+UjzDW1Uib1DqfQ1F8+eh6vVWktqXVJXPPvuMBx54gOjoaJ5++mmaNUt+pb3xJU833K1zPa2lqpclC1ffw1JfBmZMVvBwx+p234OfhYeH8/777xMWFsbSpUupW7euv0PKdbz5CxjOlWcVI1KYZ4xfzFj1K19uPJyh+zxxFYX7TMZzL+LXtm1b6tWrx4MPPmhF/PzEUx9FP5xLWiuJyJduiwoDJ30dmDHeWrLrT46ejqFJxYy7Z6F8iQKUK1GAgh7GgTC+sX//fkaNGsVdd93FsGHDGDFihL9DyvU8ped1wAmcG+Wmus0/A9hY2SZLqXFDYWYMCfN3GOYaxMfH88YbbzBx4kQCAwMZMmSIv0MyLp76KA4AB4AlmReOya1eX7qXX454W0bscnuPnqHGDYUzOCKTmXbu3Mnw4cNZu3YtXbt2Zdq0aYSEhPg7LOPiqelppaq2FZFoLq/RJICqqtUmMBnmzWX7KJgvkOsK5bvqba8rlI92Na73QVQmsxw4cIBff/2Vjz76iP79+1sRvyzGU9NTO9e/6RqPwpir1b9peR7tbEN+5hbr169n8+bNjBo1iq5du7J//34KF7Yzw6wo1QvE3e7GLgcEqmo80AK4G6ckuDHGXLXz588zfvx4mjdvzosvvkhMjDNOhyWJrMubqmVfASoiVYD/wykM+JFPozLG5EgrVqygXr16/Oc//2HUqFFs2rTJivhlA95clJygqrEi0hOYoqqvi4hd9WSMuSpRUVF06NCBChUqsGzZMtq1a5f2RiZL8CZRxIlIH2Aw0MM1L4/vQjI5wfLdfxJ9FTetxVvR4Bxry5Yt1K9fn5CQEL7++mtuvvlmChQo4O+wzFXw9s7s+3DKjO93jS/xsW/DMtlZVPR5hs1ef9XblbRxpHOUY8eOMXbsWD7++GNWrFhB27Zt6dKli7/DMungzVCo23ESReL0AeB5XwZlsrdLcc51EE92rUWH0NJebRMgQkjx/L4My2QSVWXu3LmMGTOGU6dO8cwzz9CiRQt/h2WugTcj3DUH/glUdK2feB9Fdd+GZrIiVWXvn2c5fyk+1XV+P3kBgFKF81GhpF0gl9sMHjyYDz/8kGbNmjFz5kxq167t75DMNfKm6Wk2MAHYAKT+7WByhU2RJ+n51k9erZs/j9VJyi0SEhIQEUSEdu3a0bhxY8aMGUNgoH0GcgJvEsVpVV3g80hMtnAmJg6AiV1qeRyHIV9QAE0r2c37ucG+ffsYNWoUgwcPZvjw4VbELwfyJlEsE5EXgS+Bi4kzVXWrz6IyWcK5i3FXjPB2/qKTKBpVKG5jQudycXFxTJkyhaeeeop8+fJZgsjBvEkUNyX7F5zaT20yPhyTVfy07ziDZq4ltatW8wRaLZ7cbPv27QwbNoyIiAi6d+/OW2+9xY033ujvsIyPeHPVU+vMCMRkLX+cikEVHmhXleLJLlstnC+I2jcW9VNkJis4dOgQBw8eZO7cufTt29eK+OVw3lz1VAp4Diirqt1EJBRoqqrv+To44399w8pRvqTdHGVg7dq1bNmyhdGjR9OlSxf2799PoUI2Xnhu4E3T03vAh8Cjrum9wCeu+SYLeGfVfqav+jVD9xkT69wLYT8Uzblz53jqqaeYMmUKlStXJjw8nHz58lmSyEW8SRTXq+pHIvIIgKvuk10mm4VsPBRNbLzSrV6ZDN1vyUL5KFvMboLLzZYtW8aoUaPYv38/9957Ly+99BL58l39mCEme/MmUZwTkRK4Bi8SkSY4w6GaLKR0kXw8f2ddf4dhcpCoqCg6depEpUqVWLlyJW3a2PUruZU3iWI8sACoLCIrgbJAb59GZdL0aUQkn0dEAbD3zzOUKmy/8kzG2LRpEw0bNiQkJIQFCxbQtm1b8ue3M8vcLM3xKFQ1Ame0u7bAWCBUVTf7OjDj2bdb/2DnH6cJDBBq3lCE3o1tfGFzbY4ePUq/fv1o1KgRK1euBKBz586WJIzHMbMbA1GqelRVL4lIHaAncFBEJqnqyUyL0qSoyvWF+Hh0c3+HYbI5VeXDDz9k7NixnD17lueee46WLVv6OyyThXg6o5gBxAGIyE3AZOBTIMa1zPhRfIKN32AyxsCBAxk8eDA1atRg8+bNTJw4kTx5bMgZ8zdPfRRBqnrC9bw/MENVPwE+EZEtvg/NpOaLDVH8+Otx+oWV83coJptyL+LXsWNHWrRowf33329F/EyKPJ1RBIpI4qemPbDMfZnvQjKeLNjyO498voWWVUryrzusfLO5env27KFdu3bMmjULgGHDhlmlV+ORpzOKT4HlInIMuAT8ACAiVYDTmRCbATYcjCbit78Ap3Lr2yt/JaxCCd4ZEkawlfE2VyEuLo5XX32Vp59+muDgYOukNl5LNVGo6iQRWQaUAb5X1QTXojzAmMwIzsCkb3ayJfLv6waaVCzOzKFhFMjrzZXNxji2bt3K8OHD2bBhA3feeSdTp06lTJmMvUHT5FyernoqoKqrk89X1V+SrXPeV8EZiE9IoG31Urx9VyPAGQzICrCZqxUVFUVkZCSfffYZvXr1ss+QuSqe+ii+EZGXRaSliAQnzhSR8iISLiILgW6edi4inUVkt4jsE5HHUlmnr4jsFJEdIvJR+t5GznLwxDn+t/Mo/9t5lNMX4ggKEArkDaJA3iD7Azde++mnn5g2bRpAUhG/3r1722fIXDVP7RftgdtxbrJrJSKFgARgH/AtMEpVD6e2sasjfCrQAYgC1ovIfFXd6bZONeBxoJWqRovI9df6hrK7dQf+YsistUlF+QAbIMhclbNnzzJx4kTeeOMNqlSpwrBhw8iXLx8FC9r45SZ9PPVRKDDf9UiPpsA+Vd0PICJzge7ATrd1RgFTVTXa9Zp/pvO1coSNh6IZNnsdZYvl55Xe9ckX5JzwVSllVTqNdxYvXszo0aM5dOgQ999/Py+88IIV8TPXzJc9omWBSLfpKKBZsnWqA4jIjziX3P5LVb9PviMRGQ2MBihfvrxPgvW37YdPET5rHdcVzsdHo5pTukhw2hsZ4yYyMpKuXbtSpUoVVq1axU033ZT2RsZ4Ic1aT9cgpYbQ5LcTBwHVgJuBAcC7IlLsio1UZ6hqmKqGlSpVKsMDzQrGf7aFwvmC+HBkM0sS5qps2LABgHLlyrFw4UI2b95sScJkKF8miijA/dbhEOD3FNb5WlVjVfUAsBsnceQ6J8/H0rpaKUKK22hyxjtHjhyhT58+hIWFJRXx69ChA8HB9kPDZCyPiUJEAq+hXMd6oJqIVBKRvDhlQJL3d3yFU5kWEbkOpylqfzpfL8tS1bQfV5xsGZMyVWXOnDmEhoayYMECXnjhBSviZ3zKYx+Fqsa7Ll0t6+kKp1S2jRORB4BFOP0Ps1R1h4hMAiJUdb5rWUcR2QnEA4+41ZfKERISlFv+s4LfTqR9u0lAgF22aNLWv39/Pv30U1q1asW7775LzZo1/R2SyeG86cy+DtglImuAc4kzVbVnWhuq6kJgYbJ5/3R7rsDDrkeOFK/KbyfO07JKSZpWKpHqeoLQNYOHMjU5h3sRvy5dutC6dWvuu+8+AgJ82XpsjMObRPGSz6PIBVpWKckDt+TK7hdzjX755RdGjhzJ0KFDGTlyJOHh4f4OyeQy3oxwtxTYglPjKQ+wxTXPGONDsbGxvPDCC9SvX5+dO3dSqJDdT2P8I81EISK9gI3AYGAIECEid/o6MGNys82bN9O0aVMmTpzIHXfcwc6dO+nfv7+/wzK5lDdNT/8EmqjqUQARKQ0sBub5MjBjcrMjR45w5MgRvvjiC3r2TLM70Bif8iZRBCQmCZdj+Pb+C2NypdWrV7N161buu+8+OncWSciGAAAgAElEQVTuzK+//kqBAnZfjfE/b77wF4vIQhG5S0TuwrkXYpGP4zIm1zhz5gwPPPAArVu3ZsqUKVy8eBHAkoTJMrxJFOOB93CK/DUD5gCP+DAmY3KNRYsWUadOHd566y3Gjh3Lxo0brYifyXLSbHpy3evwqethrlKC2h3XJmWRkZF069aNqlWrsnr1aru72mRZ1tfgQ6rKv+bvAKDSdXZpo3E+E+vWrQOcIn7fffcdmzZtsiRhsjRLFD6iqjyzYCcfr4vk/nZV7K5rwx9//EGvXr1o1qxZUhG/W2+91Yr4mSzPq0QhInlFpKqvg8lJXv5+N+/99BsjbqrE+I41/B2O8SNVZfbs2YSGhvLdd9/x8ssv06pVK3+HZYzXvLnhriuwDfifa7qBiNg9FB6cuxjHtJW/0r3BjTzZtZaNUZzL9e3bl+HDh1O3bl22bNnChAkTCAry5ZhhxmQsbz6tk3CudloOoKqb7ezCs7gEpwO7XkgxSxK5VHx8PCJCQEAAt99+O7fccgt33323FfEz2ZI3iSJWVU8m+8KzS3lSsPfoGU6cu8S5i3H+DsX40a5duxgxYgTDhg1j1KhRDBkyxN8hGXNNvEkUu0SkLxAgIpWAscDPvg0r+zl5/hKdpqwiwS2FFsoX6L+ATKaLjY3l5Zdf5tlnn6VQoUIULVrU3yEZkyG8SRQP4NR7SgC+xLkr+3FfBpUdXYiNJ0FhVOtKtKt5PXkCA2hQ7orhv00OtWnTJoYOHcrWrVvp168fr7/+Otdff72/wzImQ3iTKDqp6qPAo4kzRKQnTtIwyVQpVYiWVa7zdxgmkx09epTjx4/z1Vdf0b17d3+HY0yG8qZn7ckU5k3M6ECMyW5WrVrF1KlTAejcuTP79u2zJGFypFTPKESkE9AZKCsir7otKoLTDGVMrnT69Gkee+wx3n77bapXr87IkSPJly8f+fPn93doxviEpzOKP4HtQAyww+2xGLjN96EZk/UsXLiQ2rVrM336dB5++GEr4mdyhVTPKFR1E7BJRD5U1ZhMjCnbOB0TS9fXf+Dkudik4n8Bdt9EjhUZGUn37t2pUaMGn3/+Oc2aNfN3SMZkCm86s8uKyPNAKJBUlEZVq/ssqmzi2JmLRP51gbbVS1GlVCHyBgXQrqZd6ZKTqCpr166lefPmlCtXjsWLF9OqVSvy5s3r79CMyTTedGa/B8wGBKfJ6VNgrg9jynZ6NirLP28P5bHbalKqsDVD5BS///47PXr0oEWLFklF/Nq1a2dJwuQ63pxRFFDVRSIyWVV/BZ4UkR98HVhWdSYmlns/2MjpmFhiYuP9HY7xAVVl5syZjB8/nosXLzJ58mQr4mdyNW8SxUVx6nf8KiL3AIeBXNu+cvDEeVbvO07dskUpWyw/la8rRFjFEv4Oy2Sg3r178+WXX9K2bVveffddqla10mYmd/MmUTwEFALGAM8DRYHhvgwqOxjTvhodQkv7OwyTQdyL+PXo0YOOHTsyatQoK+JnDN4NhbrW9fQMMBhAREJ8GZS/HD97kcmLdntsUjp5ITYTIzKZYfv27YwcOZIRI0YwatQoBg8e7O+QjMlSPCYKEWkClAVWq+pxEamNU8rjFiDHJYv1B/5i7vpIyhQNJm9Q6r8ka95QmGrX29Cm2d2lS5d48cUXef755ylatCjFixf3d0jGZEme7sx+EegFbMHpwJ6HUzn2ZeCezAnPP2YPa0LNG4r4OwzjQxs2bGDo0KFs376dgQMHMmXKFEqVKuXvsIzJkjydUXQH6qvqBREpAfzumt6dOaH5zl/nLjF3/SHi4i8fVmPP0TN+ishkthMnTnDy5EkWLFhAt27d/B2OMVmap0QRo6oXAFT1LxH5JSckCYDvtv/BK9+n/FYKBwdxXSG7FyInWr58Odu2bWPMmDF07NiRvXv3EhwcnPaGxuRynhJFZRFJLCUuQEW3aVS1p08j86EE1+hC655oT8lkSUGAgAArw5GTnDp1igkTJjBjxgxq1qzJ3XffTb58+SxJGOMlT4miV7LpN30ZiK+duxjHD3uPEZegbDt8CnASQqAlhRxtwYIF3HPPPRw5coTx48fzzDPPWBE/Y66Sp6KASzMzEF/7LCKSfy3YmTSdNyiA/HlsqNKcLDIykl69elGzZk2++uormjRp4u+QjMmWvLnhLke4GOcMoTH/gVbkzxNIsQJ5KZgv17z9XENVWbNmDS1btkwq4teyZUurz2TMNfDpbaci0llEdovIPhF5zMN6vUVERSTMl/EAVL2+ENVKF7bifTlQVFQUd9xxB61atUoq4nfzzTdbkjDmGnmdKETkqr5ZRSQQmIpTcTYUGCAioSmsVxinPMja5MuM8UZCQgLTp08nNDSUpUuX8uqrr3LTTTf5Oyxjcow0E4WINBWRbcBe13R9EXnDi303Bfap6n5VvYRTmjylAYWfBV7BGUnPmKvWq1cv7rnnHpo0acL27dt56KGHCAy0/idjMoo3ZxSvA92AEwCqugVo58V2ZYFIt+ko17wkItIQKKeq33jakYiMFpEIEYk4duyYFy9tcrq4uDgSEpx+p169evHOO++wZMkSKleu7OfIjMl5vEkUAap6MNk8bwZiSOm606RboUUkAHgNGJfWjlR1hqqGqWqYlVkwW7dupUWLFrzzzjsA3HXXXYwcORKxYWiN8QlvEkWkiDQFVEQCReRBYI8X20UB5dymQ3DKgCQqDNQBVojIb0BzYH5mdGib7OnixYs8/fTTNG7cmIMHD1ptJmMyiTfXh96L0/xUHjgKLHHNS8t6oJqIVMIZ7Kg/MDBxoaqeAq5LnBaRFcB4VY3wNniTe6xfv56hQ4eyc+dOBg8ezGuvvUbJkiX9HZYxuYI3iSJOVftf7Y5VNU5EHgAWAYHALFXdISKTgAhVnX+1+zS5V3R0NGfPnmXhwoXcdttt/g7HmFzFm0SxXkR2A58AX6qq1yVWVXUhsDDZvH+msu7N3u7X5A7Lli1j27ZtjB07lo4dO7Jnzx4rv2GMH6TZR6GqVYDngMbANhH5SkSu+gzDGG+dPHmSUaNG0b59e6ZPn87FixcBLEkY4yde3XCnqj+p6higEXAa+NCnUZlc6+uvvyY0NJRZs2YxYcIENmzYYAnCGD9Ls+lJRArh3CjXH6gFfA209HFcJhc6dOgQffr0oVatWsyfP5+wMLsAzpiswJs+iu3AAuAVVf3Bx/GYXEZVWb16Na1bt6Z8+fIsWbKE5s2bW30mY7IQb5qeKqvqPyxJmIx26NAhunbtSps2bZKK+LVp08aShDFZTKpnFCLyH1UdB3whIpp8eXYe4c74V0JCAtOmTePRRx9FVXn99detiJ8xWZinpqdPXP9m65HtTNbTs2dPvv76azp06MCMGTOoWLGiv0MyxnjgaYS7da6ntVT1smThupEuR42AZ3wrLi6OgIAAAgIC6NevH927d2fo0KFWn8mYbMCbPorhKcwbkdGBmJxry5YtNGvWjBkzZgAwYMAAhg0bZknCmGzCUx9FP5xLYiuJyJduiwoDJ30dmMn+YmJieO6553j55ZcpUaIEN9xwg79DMsakg6c+inU4Y1CE4IxUl+gMsMmXQWWUmNh4In6LJl6V/cfO+TucXGXdunWEh4fzyy+/EB4ezquvvkqJEiX8HZYxJh089VEcAA7gVIvNlub89BsvfvdL0nS+oACCAnw6TLhxOX36NBcuXOD777+nU6dO/g7HGHMNPDU9rVTVtiISjduAQzgDEqmqZvmfh+cvOeMrfXGvcyP59YXzkTfIEoWvLF68mB07dvDQQw9x6623snv3biu/YUwO4OlbM3G40+uAUm6PxOlso3GF4jSuUJxyJQr4O5QcKTo6mmHDhtGpUydmzpxpRfyMyWFSTRSqmuB6Wg4IVNV4oAVwN1AwE2Iz2cCXX35JaGgo77//Po8//jgRERGWIIzJYbxph/kKZxjUKsD/4RQG/MinUZls4dChQ/Tv358yZcqwfv16XnjhBYKDg/0dljEmg3mTKBJUNRboCUxR1X8AZX0blsmqVDWpLlP58uVZtmwZa9eupWHDhn6OzBjjK14NhSoifYDBQA/XvDy+C+nafBoRyZNfbUdViU9Q7J6ujHPw4EHuvvtuFi1axIoVK2jbtq3VaDImF/AmUQwH7sMpM75fRCoBH/s2rPTb9+dZEhKU0W0qA1C5VCE/R5T9JSQk8NZbb/HYY48B8MYbb9C6dWs/R2WMySxpJgpV3S4iY4CqIlIT2Keqz/s+tPTLExjAhM41/R1GjtGjRw8WLFhAp06dmD59OhUqVPB3SMaYTOTNCHetgfeBwzj3UNwgIoNV9UdfB2f8JzY2lsDAQAICAhgwYAC9e/dm8ODBVp/JmFzIm87s14AuqtpKVVsCXYH/+jYs408bN26kadOmTJs2DXCK+A0ZMsSShDG5lDeJIq+q7kycUNVdgA1BlgNduHCBxx9/nKZNm3LkyBHKlSvn75CMMVmAN53ZG0VkOk7zE8AgsklRQOO9n3/+mfDwcPbs2cPw4cOZPHkyxYsX93dYxpgswJtEcQ8wBpiA00exCnjDl0GZzHfu3DliY2P53//+x6233urvcIwxWYjHRCEidYEqwDxVfSVzQjKZ5fvvv2fHjh2MGzeO9u3b88svv5A3r7UqGmMul2ofhYg8gVO+YxDwPxFJaaQ7kw2dOHGC8PBwbrvtNubMmcOlS5cALEkYY1LkqTN7EFBPVfsATYB7Myck4yuqyueff05oaCgfffQRTz75JOvXr7cEYYzxyFPT00VVPQegqsdExAZyyOYOHTrEwIEDqVevHosXL6Z+/fr+DskYkw14ShSV3cbKFqCK+9jZqtrTp5GZDKGqLF++nFtuuYUKFSqwYsUKmjZtSlCQN9cxGGOM50TRK9n0m74MxGS8AwcOMHr0aJYsWZJUxK9ly5b+DssYk814GjN7aWYGYjJOfHw8b775Jk888QSBgYG8/fbbVsTPGJNu1v6QA3Xv3p1vv/2WLl26MG3aNLvD2hhzTSxR5BDuRfwGDx7MgAEDGDhwoNVnMsZcM6+vZBKRqx4IWUQ6i8huEdknIo+lsPxhEdkpIltFZKmIWP3qdIiIiCAsLIy3334bgH79+jFo0CBLEsaYDJFmohCRpiKyDdjrmq4vImmW8BCRQGAqcBsQCgwQkdBkq20CwlS1HvA5kO67v09diOXU+VguxsandxfZzoULF3j00Udp1qwZx44ds3EijDE+4U3T0+tAN5y7tFHVLSLSzovtmuIMcrQfQETmAt0B90q0y93W/xm4y8u4L/PWin288v3upOnCwTm/RW3NmjWEh4ezd+9eRo4cyb///W+KFSvm77CMMTmQN9+oAap6MFkzhjc/28sCkW7TUUAzD+uPAL5LaYGIjAZGA5QvX/6K5b+fvECBvIGM71gDgGqlc/7wpxcuXCAhIYElS5bQvn17f4djjMnBvEkUkSLSFFBXc9I/gD1ebJdSA7mmuKLIXUAY0Dal5ao6A5gBEBYWluI+8ucJZPhNlbwIK/tauHAhO3bs4JFHHuGWW25h165d5MmTx99hGWNyOG86s+8FHgbKA0eB5nhX9ykKcL8uMwT4PflKInIrMBG4Q1UverHfXOf48ePcdddddO3alQ8//DCpiJ8lCWNMZkgzUajqn6raX1Wvcz36q+pxL/a9HqgmIpVEJC/QH5jvvoKINASm4ySJP9PzBnIyVWXu3LnUqlWLTz/9lKeffpp169ZZET9jTKZKs+lJRN4hhSYjVR3taTtVjRORB4BFQCAwS1V3iMgkIEJV5wP/BgoBn7n6QA6p6h1X/zZypkOHDhEeHk79+vWZOXMmdevW9XdIxphcyJs+iiVuz4OBO7m8kzpVqroQWJhs3j/dnttQasmoKkuXLuXWW2+lQoUKrFy5kiZNmhAYGOjv0IwxuZQ3TU+fuD3mAD1x7oswGezXX3+lffv2dOjQgZUrVwLQvHlzSxLGGL9KzxgTlQC7sysDxcfH8+qrr1K3bl02bNjA9OnTrYifMSbL8KaPIpq/+ygCgL+AK8pxmPS7/fbb+e677+jWrRtvv/02ISEh/g7JGGOSeEwU4vQw1wcOu2YlqGqK9zGYq3Pp0iWCgoIICAhg6NChDB48mP79+1t9JmNMluOx6cmVFOaparzrYUkiA6xbt47GjRvz1ltvAdC3b18GDBhgScIYkyV500exTkQa+TySXOD8+fOMGzeOFi1aEB0dTZUqVfwdkjHGpCnVpicRCVLVOOAmYJSI/AqcwynNoapqyeMqrF69mvDwcPbv38/dd9/Nyy+/TNGiRf0dljHGpMlTH8U6oBHQI5NiydESBxZavnw5N998s7/DMcYYr3lKFAKgqr9mUiw5zoIFC9i1axcTJkygXbt27Ny5k6CgnF8C3RiTs3j61iolIg+ntlBVX/VBPDnCsWPHGDt2LB9//DENGjTgwQcfJG/evJYkjDHZkqfO7ECcOkyFU3mYZFSVjz76iFq1avH5558zadIk1q5da0X8jDHZmqefuH+o6qRMiyQHOHToEMOGDaNhw4bMnDmT2rVr+zskY4y5Zp7OKOyifi8kJCSwaNEiACpUqMAPP/zAjz/+aEnCGJNjeEoUNr5mGvbu3cstt9xC586dWbVqFQBNmza1In7GmBwl1UShqn9lZiDZSVxcHP/+97+pV68emzdvZubMmVbEzxiTY9llOOnQrVs3Fi1aRPfu3Xnrrbe48cYb/R2S8aPY2FiioqKIiYnxdyjGEBwcTEhISIYOlWyJwksXL14kT548BAQEMHLkSIYPH06fPn2sPpMhKiqKwoULU7FiRfs8GL9SVU6cOEFUVBSVKlXKsP2mZzyKXOfnn3+mUaNGTJ06FYDevXvTt29f+1IwAMTExFCyZEn7PBi/ExFKliyZ4We3lig8OHfuHA899BAtW7bkzJkzVKtWzd8hmSzKkoTJKnzxWbSmp1T88MMPhIeHc+DAAe677z5efPFFihQp4u+wjDEm09kZRSri4uLIkycPK1euZOrUqZYkTJYWGBhIgwYNqFOnDrfffjsnT55MWrZjxw5uueUWqlevTrVq1Xj22WdxH1rmu+++IywsjFq1alGzZk3Gjx/vj7fg0aZNmxg5cqS/w/DoxRdfpGrVqtSoUSPp3qrkWrduTYMGDWjQoAE33ngjPXo4NVdXrFhB0aJFk5ZNmvT3vc4nT56kd+/e1KxZk1q1arFmzRoAxo8fz7Jly3z/xsDp/MhOj8aNG2tyE+dt1UaTFl8x/2rNmzdPX3jhhaTp2NjYa96nyfl27tzp7xC0YMGCSc+HDBmizz33nKqqnj9/XitXrqyLFi1SVdVz585p586d9c0331RV1W3btmnlypV1165dqup85qdOnZqhsWXE31Hv3r118+bNmfqaV2PHjh1ar149jYmJ0f3792vlypU1Li7O4zY9e/bUOXPmqKrq8uXLtWvXrimuN2TIEH3nnXdUVfXixYsaHR2tqqq//fabdujQIcVtUvpMAhGazu9da3oCjh49yj/+8Q8+++wzGjVqxLhx46yIn0mXZxbsYOfvpzN0n6E3FuHp272/079FixZs3boVgI8++ohWrVrRsWNHAAoUKMCbb77JzTffzP33388rr7zCxIkTqVmzJgBBQUHcd999V+zz7Nmz/OMf/yAiIgIR4emnn6ZXr14UKlSIs2fPAvD555/zzTff8N577zF06FBKlCjBpk2baNCgAfPmzWPz5s0UK1YMgKpVq/Ljjz8SEBDAPffcw6FDhwCYMmUKrVq1uuy1z5w5w9atW6lfvz7gjBD54IMPcuHCBfLnz8/s2bOpUaMG7733Ht9++y0xMTGcO3eOZcuW8e9//5tPP/2Uixcvcuedd/LMM88A0KNHDyIjI4mJiWHs2LGMHj3a6+Obkq+//pr+/fuTL18+KlWqRNWqVVm3bh0tWrRIcf0zZ86wbNkyZs+e7XG/p0+fZtWqVbz33nsA5M2bN6l2XIUKFThx4gRHjhzhhhtuuKb405KrvwlVlQ8++IAHH3yQs2fP8vzzz/PII49k6PXHxmSm+Ph4li5dyogRIwCn2alx48aXrVOlShXOnj3L6dOn2b59O+PGjUtzv88++yxFixZl27ZtAERHR6e5zZ49e1iyZAmBgYEkJCQwb948hg0bxtq1a6lYsSKlS5dm4MCBPPTQQ9x0000cOnSITp06sWvXrsv2ExERQZ06dZKma9asyapVqwgKCmLJkiU88cQTfPHFFwCsWbOGrVu3UqJECRYvXszevXtZt24dqsodd9zBqlWraNOmDbNmzaJEiRJcuHCBJk2a0KtXL0qWLHnZ6z700EMsX778ivfVv39/HnvsscvmHT58mObNmydNh4SEcPjw4VSPzbx582jfvv1lTdpr1qyhfv363HjjjUyePJnatWuzf/9+SpUqxbBhw9iyZQuNGzfmv//9LwULFgSgUaNG/Pjjj/Tq1Sut/45rkqsTxaFDhxg5ciRhYWHMnDkz6VeVMel1Nb/8M9KFCxdo0KABv/32G40bN6ZDhw6A82MotatgrubqmCVLljB37tyk6eLFi6e5TZ8+fZLK2fTr149JkyYxbNgw5s6dS79+/ZL2u3PnzqRtTp8+zZkzZyhc+O8C1X/88QelSpVKmj516hTh4eHs3bsXESE2NjZpWYcOHShRogQAixcvZvHixTRs2BBwzor27t1LmzZteP3115k3bx4AkZGR7N2794pE8dprr3l3cOCyPp9Eno7vxx9/fFmfS6NGjTh48CCFChVi4cKF9OjRg7179xIXF8fGjRt54403aNasGWPHjuWll17i2WefBeD666/n999/9zrO9Mp1ndkJCQl89913gHPq9uOPP7Jq1SpLEiZby58/P5s3b+bgwYNcunQp6Z6f2rVrExERcdm6+/fvp1ChQhQuXJjatWuzYcOGNPefWsJxn5f82v3EX73gNIft27ePY8eO8dVXX9GzZ0/A+Xtcs2YNmzdvZvPmzRw+fPiyJJH43tz3/dRTT9GuXTu2b9/OggULLlvm/pqqyuOPP56073379jFixAhWrFjBkiVLWLNmDVu2bKFhw4Yp3nfw0EMPJXUuuz9eeumlK9YNCQkhMjIyaToqKirVig0nTpxg3bp1dO3aNWlekSJFKFSoEABdunQhNjaW48ePExISQkhICM2aNQOce7g2btyYtF1MTAz58+dP8XUyUq5KFHv27OHmm2+mS5curFy5EoCwsDAr4mdyjKJFi/L6668zefJkYmNjGTRoEKtXr2bJkiWAc+YxZswYJkyYAMAjjzzCCy+8wJ49ewDni/vVV68ck6xjx468+eabSdOJTU+lS5dm165dSU1LqRER7rzzTh5++GFq1aqV9Os9+X43b958xba1atVi3759SdOnTp2ibNmyAElt9ynp1KkTs2bNSupDOXz4MH/++SenTp2iePHiFChQgF9++YWff/45xe1fe+21pCTj/kje7ARwxx13MHfuXC5evMiBAwfYu3cvTZs2TXG/n332Gd26dSM4ODhp3pEjR5LOStatW0dCQgIlS5bkhhtuoFy5cuzevRuApUuXEhoamrTdnj17LmuW85Vsmyiios/z4dqDfLj2IHuOnvW4blxcHC+//DL16tVj27ZtzJ49mzZt2mRSpMZkroYNG1K/fn3mzp1L/vz5+frrr3nuueeoUaMGdevWpUmTJjzwwAMA1KtXjylTpjBgwABq1apFnTp1+OOPP67Y55NPPkl0dDR16tShfv36SW33L730Et26deOWW26hTJkyHuPq168fH3zwQVKzE8Drr79OREQE9erVIzQ0lGnTpl2xXc2aNTl16hRnzpwBYMKECTz++OO0atWK+Pj4VF+vY8eODBw4kBYtWlC3bl169+7NmTNn6Ny5M3FxcdSrV4+nnnrqsr6F9KpduzZ9+/YlNDSUzp07M3Xq1KQfoF26dLmseWju3LkMGDDgsu0///zzpGM7ZswY5s6dm3S29sYbbzBo0KCkIqRPPPEE4NQY27dvH2FhYdccf1okpba1rCwsLEwjIiJ49POtfBLx96lerTJF+G5syhVcO3XqxOLFi+nZsydTp071+RUCJnfZtWsXtWrV8ncYOdprr71G4cKFs/y9FJlp3rx5bNy4Mam/wl1Kn0kR2aCq6coq2bYzOzY+gRuLBvPV/c6ldEULXH6lUkxMDHny5CEwMJDRo0czevRon18ZYIzxjXvvvZfPPvvM32FkKXFxcV5dsZYRsm3TE0BgoHB9kWCuLxJMvqC/+xl+/PFHGjRokNSh16tXL0sSxmRjwcHBDB482N9hZCl9+vRJui/F17J1okju7NmzjBkzhtatWxMTE2PNASbTZLcmXJNz+eKzmGMSxcqVK6lTpw5vvvkmDzzwANu3b0+6ltwYXwoODubEiROWLIzfqWs8CvcrqjJCtuujiI1Xjp6O4ULslVc7FChQgB9++OGKEgDG+FJISAhRUVEcO3bM36EYkzTCXUbKdlc95StTTcuETwGg8JENDKgelHS5WHx8vN0TYYwxKbiWq5582vQkIp1FZLeI7BORK+5SEZF8IvKJa/laEamY1j6DAoQJbUpz/bqpbJ/zNPPmzePSpUsAliSMMcYHfJYoRCQQmArcBoQCA0QkNNlqI4BoVa0KvAa8nNZ+Ey6cYeKgDmz9aRkvvvgiP/30U1I1RWOMMRnPl2cUTYF9qrpfVS8Bc4HuydbpDsxxPf8caC9pVCqLiT5KnTp12LJlC4899phVejXGGB/zZWd2WSDSbToKaJbaOqoaJyKngJLAcfeVRGQ0kFgw/uLq1au3WxE/AK4j2bHKxexY/M2Oxd/sWPytRno39GWiSOnMIHnPuTfroKozgBkAIhKR3g6ZnMaOxd/sWPzNjsXf7Fj8TUQi0l4rZb5seooCyrlNhwDJC6cnrSMiQUBR4C8fxmSMMeYq+V6svX8AAAosSURBVDJRrAeqiUglEckL9AfmJ1tnPhDuet4bWKbZ7XpdY4zJ4XzW9OTqc3gAWAQEArNUdYeITMIZ5Hs+MBN4X0T24ZxJ9Pdi1zN8FXM2ZMfib3Ys/mbH4m92LP6W7mOR7W64M8YYk7lyTK0nY4wxvmGJwhhjjEdZNlH4ovxHduXFsXhYRHaKyFYRWSoiFfwRZ2ZI61i4rddbRFREcuylkd4cCxHp6/ps7BCRjzI7xszixd9IeRFZLiKbXH8nXfwRp6+JyCwR+VNEtqeyXETkdddx2ioijbzasapmuQdO5/evQGUgL7AFCE22zn3ANNfz/sAn/o7bj8eiHVDA9fze3HwsXOsVBlYBPwNh/o7bj5+LasAmoLhr+np/x+3HYzEDuNf1PBT4zd9x++hYtAEaAdtTWd4F+A7nHrbmwFpv9ptVzyh8Uv4jm0rzWKjqclU975r8GeeelZzIm88FwLPAK0BMZgaXybw5FqOAqaoaDaCqf2ZyjJnFm2OhQBHX86JceU9XjqCqq/B8L1p34P/U8TNQTETKpLXfrJooUir/UTa1dVQ1Dkgs/5HTeHMs3I3A+cWQE6V5LESkIVBOVb/JzMD8wJvPRXWg+v+3d/ZBVtVlHP98Q14EUSOwoKa2BM0kohEbiMwKhylrGGpIM97HcmpMggmnIRsjp2EsQyvfoJgCJ2IQA0LSEI0XXV7UGFgEQRvEl4YpaojUiAif/nh+F85ed++ee9m9d9l9PjNn9pzfOef3e85z7p7n93LO9yepXtIWSZ+umnXVJY8vZgMTJL0CPATcUB3T2h3lPk+A9jtxUavJf3QAcl+npAnAMODyNrWodpT0haS34CrEU6plUA3J87s4A+9++gTeynxc0mAz+2cb21Zt8vjiGmChmc2VNAL/fmuwmb3R9ua1Kyp6brbXFkXIf5wkjy+QdAVwEzDGzI5WybZq05IvegODgfWS9uN9sKs66IB23v+R35nZMTN7AdiLB46ORh5fXAvcD2Bmm4EeuGBgZyPX86SY9hooQv7jJC36InW3zMeDREfth4YWfGFmh82sr5nVmVkdPl4zxswqFkNrx+T5H1mJv+iApL54V9S+qlpZHfL44iVgFICki/BA0Rnnrl0FTEpvPw0HDpvZgZZOapddT9Z28h+nHTl9cRtwFrAsjee/ZGZjamZ0G5HTF52CnL5YA4yWtBs4DtxoZv+ondVtQ05ffAv4haQZeFfLlI5YsZS0BO9q7JvGY74HdAUws3n4+MyVwJ+BfwNTc+XbAX0VBEEQtCLttespCIIgaCdEoAiCIAhKEoEiCIIgKEkEiiAIgqAkESiCIAiCkkSg6GRIOi5pe2apK3FsXXMqlGWWuT4pe+5IchIXVpDH1yRNSutTJA3I7Fsg6QOtbOdTkobmOGe6pJ6nWnZO+8ZKujmtf1zSNkn/kzSugrx6SlosaaekZyQ9IemsVrR1gKQHMttLklrpDEm3pA9Ec/lPUjdJG9OHtUEtqLXaYSzVXYDXyji2jmZUKMsscz1JxRW4DljVWvm1sm+ydk4F1uY4Zz/Qtw1sOaOJtE2FstK9GQLcB4yrIP9ZwO2Z7QuB7q19HSnvdwAvnor/8O8BxreFfbG0vESLIii0HB5PNdRtkj7axDEXS3oytUIaJA1K6RMy6fMldWmhuI3AwHTuKPn8ADvlOvrdU/qtOjm/xo9T2mxJM1PteRiwOJV5ZmoJDJP0dUk/ytg8RdKdFdq5mYxYmqR7JT0tn9fh+yltGjAAWCdpXUobLWlz8uOypmrpkobKRfoaJK2Q9NaUvl7SHEkbgG8WnXMBcNTM/g5gZvvNrAGoVKuoP/CXwoaZ7TWzo+m3sEfSomTfA4Uav6RLJG2Q9CdJa5RURyUNlPRoaoltk3R+UWv0EeC85PvLJC2UzxfSyH+SrpV0R+aavyrp9rS5Ehhf4bUGp0qtI1Us1V3wL3S3p2VFSusJ9Ejrg/CvWSHTogDuJNXocM3/M4GLgAeBrin9HmBSE2Wu52RN/UZgKS6h8DJwQUq/D5gO9ME1iQofg56b/s4GZhbnl90G+uFy04X0h4GPVWjndGBOZl+f9LdLOm5I2t7PyVp+XzwQ9krb3wZubqKcBuDytH4L8JNM+fc0c9+mAnObSF9IZS2KocDf8ID4A2BQ5p4bMDJt/xKYiX/duwnol9Kvxr+ABtgKfD6t98B/T9nfzon1YpuL/NcLn1eicJ82AR/M+P1grf9/OusSfX6djyNmVtz33hW4K/XJH8c1gYrZDNwk6V3AcjN7XtIo4BLgKbl0yJn4w6cpFks6gj8YbsC7Ol4ws+fS/kXA9cBd+DwSCyT9HsgtF25mByXtk2vYPJ/KqE/5lmNnL/zBlJ396ypJ1+GyN/3xyW8ais4dntLrUzndcL+dQNI5ePDbkLnuZZlDljZjV39aUZvIzLZLeh8wGrgC980I4AjwspnVp0N/DUwD/oALLq5N19YFOCCpN/BOM1uR8v0PgCqYGsbMXpf0R+Bzkp7FA8bOtO+4pP9K6m1mr1Z84UFFRKAIAGYAfwU+hL/g8KYJf8zsN5K2Ap8F1kj6Ci5ZvMjMZuUoY7xlxPkkNTl3iLluz0dwAbcvAd8APlXGtSwFrgL24C0mkz+1ctuJz5B2K3A38AVJ78Vr1Zea2SFJC/GaczHCxzWuKcPeYl5vJv0IrpCcG0nX45MXAVxpZo1UQs3sNWA5sFzSG7gG0G95s+y04de2y8xGFJVxNq3LAuA7+P37VdG+7nTsyajaLTFGEYA/gA6Ya/NPxGuLjUi1z31m9jNcgXII8BgwTtJ56Zg+yj9f9x6gTtLAtD0R2JD69M8xs4fw7p+m3jx6FZcUb4rlwFh8/oFC7bwsO83sGPBdYLhcafRs/AF+WNLbgc80Y8sWYGThmuRvFjVqnZnZYeCQpMuy192cLRmeJY3t5MXM7jazoWlpFCQkjcyMjXTDW0Ivpt3vTq0LcD8+gXcH9iukS+oq6WIz+xfwiqSxKb27ynsLrNG9NLOtuAz2l4ElGXvfhnc9HSsj76CViEARgPfZT5a0Be92aqpWezXwjKTtwPvx6RR34w/URyQ1AGvxLpIWSV0UU3HF2534oOw8/KGxOuW3AW/tFLMQmFcYzC7K9xCwG3iPmT2Z0sq208yOAHPxcZEd+NzTu/A++/rMoT8HHpa0zswO4pMmLUnlbMF9Vcxk4LZ0zFB8nKIlNgIfTq0jJF0qVwf9IjBf0q4ceWQ5Hw/MO9O1PY23JsCD0uRkXx/gXvMpRscBP5S0Ax/jKrz0MBGYlo7fhL/llJcT/suk3Q/Up3tZ4JO48mlQA0I9NghOEyT9FHjQzB5twzLqgNVmNrityshhw2rgDjN7LJO2HJhlZntrZVdnJloUQXD6MAd/o6hDIulcSc/hL1xkg0Q3YGUEidoRLYogCIKgJNGiCIIgCEoSgSIIgiAoSQSKIAiCoCQRKIIgCIKSRKAIgiAISvJ/8PjiuhwXubAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_roc_curve(target_test, target_predicted_proba)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the area under ROC curve is 0.756 which is very similar to the accuracy (0.732). However the ROC-AUC score of a random model is expected to 0.5 on average while the accuracy score of a random model depends on the class imbalance of the data. ROC-AUC can be seen as a way to callibrate the predictive accuracy of a model against class imbalance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We previously decided to randomly split the data to evaluate the model on 20% of held-out data. However the location randomness of the split might have a significant impact in the estimated accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7318435754189944"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_train, features_test, target_train, target_test = train_test_split(\n",
    "    features_array, target, test_size=0.20, random_state=0)\n",
    "\n",
    "logreg.fit(features_train, target_train).score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6703910614525139"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_train, features_test, target_train, target_test = train_test_split(\n",
    "    features_array, target, test_size=0.20, random_state=1)\n",
    "\n",
    "logreg.fit(features_train, target_train).score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.664804469273743"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_train, features_test, target_train, target_test = train_test_split(\n",
    "    features_array, target, test_size=0.20, random_state=2)\n",
    "\n",
    "logreg.fit(features_train, target_train).score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So instead of using a single train / test split, we can use a group of them and compute the min, max and mean scores as an estimation of the real test score while not underestimating the variability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.63128492, 0.68715084, 0.70224719, 0.73033708, 0.71751412])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "scores = cross_val_score(logreg, features_array, target, cv=5)\n",
    "scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.6312849162011173, 0.6937068296293303, 0.7303370786516854)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.min(), scores.mean(), scores.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cross_val_score` reports accuracy by default be it can also be used to report other performance metrics such as ROC-AUC or f1-score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.6109354413702239, 0.7212318165109173, 0.7877673796791443)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_val_score(logreg, features_array, target, cv=5,\n",
    "                         scoring='roc_auc')\n",
    "scores.min(), scores.mean(), scores.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**:\n",
    "\n",
    "- Compute cross-validated scores for other classification metrics ('precision', 'recall', 'f1', 'accuracy'...).\n",
    "\n",
    "- Change the number of cross-validation folds between 3 and 10: what is the impact on the mean score? on the processing time?\n",
    "\n",
    "Hints:\n",
    "\n",
    "The list of classification metrics is available in the online documentation:\n",
    "\n",
    "  http://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values\n",
    "  \n",
    "You can use the `%%time` cell magic on the first line of an IPython cell to measure the time of the execution of the cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2857142857142857 0.4158823529411765 0.5588235294117647 0.08732661354820218\n",
      "Wall time: 34 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "scores = cross_val_score(logreg, features_array, target, cv=10, scoring='recall')\n",
    "\n",
    "print(scores.min(), scores.mean(), scores.max(), scores.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More feature engineering and richer models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now try to build richer models by including more features as potential predictors for our model.\n",
    "\n",
    "Categorical variables such as `data.Embarked` or `data.Sex` can be converted as boolean indicators features also known as dummy variables or one-hot-encoded features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Sex_male</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sex_female  Sex_male\n",
       "0           0         1\n",
       "1           1         0\n",
       "2           1         0\n",
       "3           1         0\n",
       "4           0         1"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#one hot encoding of the sex variables\n",
    "pd.get_dummies(data.Sex, prefix='Sex').head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Embarked_C  Embarked_Q  Embarked_S\n",
       "0           0           0           1\n",
       "1           1           0           0\n",
       "2           0           0           1\n",
       "3           0           0           1\n",
       "4           0           0           1"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.get_dummies(data.Embarked, prefix='Embarked').head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can combine those new numerical features with the previous features using `pandas.concat` along `axis=1`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Sex_male</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.2500</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>71.2833</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.9250</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53.1000</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.0500</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass   Age  Sex_female  Sex_male  Embarked_C  Embarked_Q  \\\n",
       "0   7.2500       3  22.0           0         1           0           0   \n",
       "1  71.2833       1  38.0           1         0           1           0   \n",
       "2   7.9250       3  26.0           1         0           0           0   \n",
       "3  53.1000       1  35.0           1         0           0           0   \n",
       "4   8.0500       3  35.0           0         1           0           0   \n",
       "\n",
       "   Embarked_S  \n",
       "0           1  \n",
       "1           0  \n",
       "2           1  \n",
       "3           1  \n",
       "4           1  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rich_features = pd.concat([data.get(['Fare', 'Pclass', 'Age']),\n",
    "                           pd.get_dummies(data.Sex, prefix='Sex'),\n",
    "                           pd.get_dummies(data.Embarked, prefix='Embarked')],\n",
    "                          axis=1)\n",
    "rich_features.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By construction the new `Sex_male` feature is redundant with `Sex_female`. Let us drop it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.2500</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>71.2833</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.9250</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53.1000</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.0500</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass   Age  Sex_female  Embarked_C  Embarked_Q  Embarked_S\n",
       "0   7.2500       3  22.0           0           0           0           1\n",
       "1  71.2833       1  38.0           1           1           0           0\n",
       "2   7.9250       3  26.0           1           0           0           1\n",
       "3  53.1000       1  35.0           1           0           0           1\n",
       "4   8.0500       3  35.0           0           0           0           1"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rich_features_no_male = rich_features.drop('Sex_male', 1)\n",
    "rich_features_no_male.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fare          891\n",
       "Pclass        891\n",
       "Age           714\n",
       "Sex_female    891\n",
       "Embarked_C    891\n",
       "Embarked_Q    891\n",
       "Embarked_S    891\n",
       "dtype: int64"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rich_features_no_male.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us not forget to imput the median age for passengers without age information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.2500</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>71.2833</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.9250</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53.1000</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.0500</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass   Age  Sex_female  Embarked_C  Embarked_Q  Embarked_S\n",
       "0   7.2500       3  22.0           0           0           0           1\n",
       "1  71.2833       1  38.0           1           1           0           0\n",
       "2   7.9250       3  26.0           1           0           0           1\n",
       "3  53.1000       1  35.0           1           0           0           1\n",
       "4   8.0500       3  35.0           0           0           0           1"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rich_features_final = rich_features_no_male.fillna(rich_features_no_male.dropna().median())\n",
    "rich_features_final.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can finally cross-validate a logistic regression model on this new data an observe that the mean score has significantly increased:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic Regression CV scores:\n",
      "min: 0.770, mean: 0.786, max: 0.810\n",
      "Wall time: 29 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "logreg = LogisticRegression(C=1, solver='liblinear')\n",
    "scores = cross_val_score(logreg, rich_features_final, target, cv=5, scoring='accuracy')\n",
    "print(\"Logistic Regression CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**:\n",
    "\n",
    "- change the value of the parameter `C`. Does it have an impact on the score?\n",
    "\n",
    "- fit a new instance of the logistic regression model on the full dataset.\n",
    "\n",
    "- plot the weights for the features of this newly fitted logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%load solutions/04A_plot_logistic_regression_weights.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEbCAYAAAAibQiyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHA9JREFUeJzt3Xm4HVWZ7/Hvm4RBkVaRoAjBgKIGFVEjMtjdcgWZriAKCtjOEhtERWltcNbbary2E7atRJFGBXG4orFBcaC9qIASrooMcok0SBqEAIp9sWUI7/1jrUN2DueQs7L32XV2+H6eZz9nV+06u9491a+q1qqqyEwkSWoxq+sCJEmjx/CQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktRsTtcFTGbzzTfP+fPnd12GJI2Uiy666KbMnDvd85mx4TF//nyWLVvWdRmSNFIi4pphzMfdVpKkZoaHJKmZ4SFJamZ4SJKaGR6SpGaGhySpmeEhSWpmeEiSms3YgwSl6TD/uDO7LmENVy/ev+sSpHXilockqZnhIUlqZnhIkpoZHpKkZoaHJKmZ4SFJamZ4SJKaGR6SpGaGhySpmeEhSWpmeEiSmhkekqRmhockqZnhIUlqZnhIkpoZHpKkZgMJj4j4XETcGBGXTPJ4RMQJEbE8Ii6OiKcOYr6SpG4MasvjX4B97uPxfYHt620R8KkBzVeS1IGBhEdmngvcch+THAh8PosLgIdExJaDmLckafiG1eaxFXBtz/CKOm4NEbEoIpZFxLKVK1cOqTRJUqthhUdMMC7vNSJzSWYuzMyFc+fOHUJZkqR1MazwWAHM6xneGrhuSPOWJA3YsMJjKfDS2utqF+DWzLx+SPOWJA3YnEE8SUR8CXgWsHlErADeBWwAkJmfBs4C9gOWA38CXjGI+UqSujGQ8MjMw9byeAKvHcS8JEnd8whzSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQYSHhGxT0RcERHLI+K4CR5/eUSsjIhf1NurBzFfSVI35vT7BBExG/gksBewArgwIpZm5mXjJv1yZh7d7/wkSd0bxJbHzsDyzLwqM+8ATgcOHMDzSpJmqEGEx1bAtT3DK+q48V4QERdHxNciYt4A5itJ6sggwiMmGJfjhr8FzM/MHYHvA6dM+EQRiyJiWUQsW7ly5QBKkyRNh0GExwqgd0tia+C63gky8+bMvL0OfgZ42kRPlJlLMnNhZi6cO3fuAEqTJE2HQYTHhcD2EbFtRGwIHAos7Z0gIrbsGTwAuHwA85UkdaTv3laZeVdEHA2cDcwGPpeZl0bEe4FlmbkUeH1EHADcBdwCvLzf+UqSutN3eABk5lnAWePGvbPn/vHA8YOYlySpex5hLklqZnhIkpoZHpKkZoaHJKmZ4SFJamZ4SJKaGR6SpGaGhySpmeEhSWpmeEiSmhkekqRmhockqZnhIUlqZnhIkpoZHpKkZoaHJKmZ4SFJamZ4SJKaGR6SpGaGhySpmeEhSWpmeEiSmhkekqRmhockqZnhIUlqZnhIkpoZHpKkZoaHJKmZ4SFJamZ4SJKaGR6SpGaGhySpmeEhSWpmeEiSmhkekqRmhockqdlAwiMi9omIKyJieUQcN8HjG0XEl+vjP42I+YOYrySpG32HR0TMBj4J7AvsABwWETuMm+xVwO8z8zHAR4EP9jtfSVJ3BrHlsTOwPDOvysw7gNOBA8dNcyBwSr3/NeDZEREDmLckqQODCI+tgGt7hlfUcRNOk5l3AbcCDxvAvCVJHZgzgOeYaAsi12EaImIRsAhgm2226b+yETL/uDO7LmENVy/ef63TjGLNU5lmphm193nU6oXRrLlrg9jyWAHM6xneGrhusmkiYg7wYOCW8U+UmUsyc2FmLpw7d+4ASpMkTYdBhMeFwPYRsW1EbAgcCiwdN81S4GX1/sHAOZl5ry0PSdJo6Hu3VWbeFRFHA2cDs4HPZealEfFeYFlmLgVOAr4QEcspWxyH9jtfSVJ3BtHmQWaeBZw1btw7e+7/GThkEPOSJHVvIOEhSWNGobFX/TM8pBnOhbFmIs9tJUlqZnhIkpoZHpKkZoaHJKmZ4SFJamZ4SJKaGR6SpGaGhySpmeEhSWpmeEiSmhkekqRmhockqZnhIUlqZnhIkpoZHpKkZoaHJKmZ4SFJamZ4SJKaGR6SpGaGhySpmeEhSWo2p+sCJKlrVy/ev+sSRo5bHpKkZoaHJKmZ4SFJamZ4SJKaGR6SpGaGhySpmeEhSWpmeEiSmhkekqRmhockqZmnJ9E685QO0v2XWx6SpGaGhySpWV/hERGbRcT3IuLK+vehk0y3KiJ+UW9L+5mnJKl7/W55HAf8IDO3B35QhyfyX5m5U70d0Oc8JUkd6zc8DgROqfdPAZ7X5/NJkkZAv+Hx8My8HqD+3WKS6TaOiGURcUFETBowEbGoTrds5cqVfZYmSZoua+2qGxHfBx4xwUNva5jPNpl5XURsB5wTEb/KzN+MnygzlwBLABYuXJgNzy9JGqK1hkdm7jnZYxFxQ0RsmZnXR8SWwI2TPMd19e9VEfFD4CnAvcJDkjQa+t1ttRR4Wb3/MuCb4yeIiIdGxEb1/ubA7sBlfc5XktShfsNjMbBXRFwJ7FWHiYiFEfHZOs0CYFlE/BL4N2BxZhoekjTC+jo9SWbeDDx7gvHLgFfX++cBT+pnPpKkmcUjzCVJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDUzPCRJzQwPSVIzw0OS1MzwkCQ1MzwkSc0MD0lSM8NDktTM8JAkNTM8JEnNDA9JUjPDQ5LUzPCQJDWb088/R8QhwLuBBcDOmblskun2AT4OzAY+m5mL+5nv+ujqxft3XYIkTVm/Wx6XAM8Hzp1sgoiYDXwS2BfYATgsInboc76SpA71teWRmZcDRMR9TbYzsDwzr6rTng4cCFzWz7wlSd0ZRpvHVsC1PcMr6jhJ0oha65ZHRHwfeMQED70tM785hXlMtFmSk8xrEbAIYJtttpnCU0uSurDW8MjMPfucxwpgXs/w1sB1k8xrCbAEYOHChRMGjCSpe8PYbXUhsH1EbBsRGwKHAkuHMF9J0jTpKzwi4qCIWAHsCpwZEWfX8Y+MiLMAMvMu4GjgbOBy4CuZeWl/ZUuSutRvb6szgDMmGH8dsF/P8FnAWf3MS5I0c3iEuSSpWWTOzHbpiFgJXNN1HdNoc+CmrotoNGo1j1q9YM3Dsj7X/KjMnDvdxczY8FjfRcSyzFzYdR0tRq3mUasXrHlYrLl/7raSJDUzPCRJzQyP7izpuoB1MGo1j1q9YM3DYs19ss1DktTMLQ9JUjPDQ5LUzPCYQWItF0bRuqsXJfM9lgbE8JgBImIWQPY0QI3iQm5sAT2TjL2Pmbmqjtp4/GPrs5n4mUxVRGwcEcdGxEFd1zIVtd6DIuKJXdcyVRGxUUQ8td5v+j0YHjNAZt4NEBEHRMSHImIn4C86LqvZ2AI6Ig6LiAd1XQ+sDuSIeH5E/Ah4V0Qc0/vY+qhnhWTsM5kfERt1W9XU1Wv7/Ax4PDA/Ih7ccUn3KSKOBJYBhwBfiYgPRMSuHZd1nyLiCMpZz18ZEXNafw+GR0fGftz1/gYR8XHgbylnHn4H8LKuapuqsdcwtsYSEQdHxAXAc+lZw++qrp7hnYGXUi40dhFwTETs1kVt0y0ing5rrJAcEhGXAW8BPj/+vZmJImI7ynfoZZl5RGZ+NDNv7bquyUTE44EDgMMz83BKgNwIfKDTwu5DRHwYOBg4JDOPBu5ufY4Z/0VaX2Xm3RGxaUTsn5l3llG5H/BgYAHw624rvG8RMXtsAcXqq0XuB7w7Mw/PzJsioq+zNq9LTXDPe7tBROxYF5ZPA84BngW8FXh/Zp43zNqGISIeDSwe280TEdsD/x3YC/iflIXagd1VOGV/BczOzJ9HxKwRCLzdgbsy8+KIiHrJiZOAWyPiLR3XtoYoNqD8Zt+WmVdExKb1N7Nhy3MN9cet1SJif+AlwI0R8VPgGRHx78A3gN0y8w8R8Qjghpm4eyUzV0XEw4DFwKUR8TlgFfCKiNibcsXI6yPipMz85bBqAoiIlwDPp3y//wZYTrkkwCeAXTLz9oiYC2yxPlxbpi6wknKFztOAQyPiW8AWlCt5vhVYCLyqXkZhplsF/LiuoKxa69TdWwWcX3f93AWQmX+MiJOB/SJiw8y8o9sSi8zMuqPgscBfRcQLgc3qbsG5EfGazLxiKs810xN95NWkH78bZQHwXuDGzHw9cCdwPvDlzHxjDY6DgVcDmwy96AlMsIvq6ZSarwVOy8w/UhbOPwC+BpwMbMialyAedE17RMS2PcMPiYhTgb2BLwNPoqxtX0lZqN5Qg2PPWuOC6aptGHq2tLL+/S/K+/9n4HBKcBxOORPrbpl5ckRsFxHPqv8/UzsM3E7ZSnwo3PMbmh0RD4uIvxwb12F94/2JUu9DYI3dppsDs2ZKcECpre7peB8wl7JScTHwr8C/A0dFxAOn9GSZ6W0IN2A7ykJtbh0+Hji/5/G9KAu0LwJfBS4A9psBdc+i/ADGj38dcBTwwPranjnu8d0oDZ57TlNdmwHXA+dS1qih/Hi/D2xeh18KnAg8hbKmdT7wv4DzgIO7fm8H+F48HzgSeHwdfh5wZv1sTgQ+BOxSv38XAv9APbtER/VuWmtaOPYdm2CasyntNJv1jHt2fT0PmIH1fhd487h6X1k/m3tNP0Nqnl3/blD/Pgb48VTf306+POvzDZhNSfXd63AAHwR+Sdl9cBFljWTzGhRH9vzvXGB/4NUz4HU8cNzwEyjn1jmM0i6ze10QfQf4GGXt9h3Ao4BPAj8FnjON9T24LiCPrGHwUuBxwAnAs3umu5Cyv39W/SweM+55OluIDuA9eDhljfFblA4W5wI71u/gJ+rC9wHAm4DT6wJ5r45rjrpgey/wqQkeH1ugPQM4hbIltT/wOeAS4IAZWu8uwOcpKy/PobR53F2/f+/pqOb3TFTzJPVvUWv+BLDxVObjbqsBiohXU748j6Z80an3/x+wM+WL9ATg45l5E/B1YN+IeCRAZq7MzDMz87NDL75HROwLHBcRW9bh11GC7jxK4/P7KCG4D3BYZh5DWcN6XGZeAyzJzGdk5nenqb7I0vvmFuBBwBuAXYFDKb28HhMRj6qTXwRsBTw6i+X1OdbY5TPTTbKbZjPgxMx8LmWtcT7woSztBF+krKnvlJkfAY7KzL0z83v1+Tr57dfP4D8pvQq3q7sQ79W1ODN/SlkxOI2yFXsD8OTMXDpD672g1vsVSnj8jrIl/FpK8HRR868nqnmcJ0XEu4EfUd7jYzLzz1OZj+ExIBGxBWXN/HWZeWhm3lp7MSwHPkpZ+30T5YdwYETslplfB35P6RHTuVjzgLKHUhbIUBqcdwX+g9IlcTvg9Zl5M7BRRLyWssZyIUAOqYEcOIOyyX0hcAVl7fvxlHaWT0bED4HbKFtDa3TNzdFoiL3XAaQR8aaIWBQRj83My4GfRMRS4GHAk4GHR8SL6sL3J5TPCsr3bI0eaUOqP8YN7xIRl1B6VG1KOcZgdpbePmtMm5l/ysyTgLdn5vFZOmlM60GPfdZ7W2YuAd6SmW/LzD9m5s9q3dPWRtNPzZT2wMsoewneWmudUi4YHgOSmTdSNq0XAETE54F31oc3oKyFHJGZ/4fSQHVqfezY+oXrzFijfs9a1LcpX6pdImLbOrwfZTfIoZS1q2dG6Rq6E6Wx8CWZ+fFh1NuztbAJsFNEnE45huM9wM3AlpTdZsdm5rHAbyiNgSOjfiaRq4/X+IuI+BvK5zAP+ExEbEbZLXUj8NrM/D2wEvhIXci+PzNPhTUa1YcWmnWB1XvWhFmUraGPZeaRwNspjft/O9n/99wfez+mrf5B1Tu2kO5dUE/XFm4/Ndff/G2Z+ZXMvKZ2SrjnO7c2hsdgvQE4LSIuBi4Fjqvj51F2p+wREX8PfI/S8EfdfdWZ+mXJ+oXfNiI+HeXI2LMoXV33qpPuBZxew28TyprugZn5ncw8JDMv7qD8f611rczMHTLzFEpHhH+hNArvHBE/p/SyOreD+tZZ/UwyIraKiC9R9qfvSllDfAdll8RxlAXD4yi9ZE6nfG4vzMxVWbuNdtUzqa7FPqhuKW1dF0oLgLEechdS2mn2joiH19c7q2chvCoiNqZsXeZ072IcVL1RuuZOe7391tyzYrJxRGxcvzNTrtnwGKDMvI2yBnxlZn5wbHO1Lli/Qunt8nTgnzJzcVd11jWMl0Tp5rphHXcgpUH1KuCSurvtEuAJEbEN8L+Bt0bECcCLKLsSPtLNK7jHHyhBsRTuWQv7dWb+qP4IVlACbkYdqDWZuHd36NdTdnd+m7ILakdKmxPAP1KCc1PKfvXHAJdmORr7J73PO4yFWG/9PcOHU3q47QAcGxHHAv8MPLkuyP4TuIPSVvOGWuvdufp4naMpnQG2st5pr3nL5mKyw54X6+ONEsgrgB3r8IY9j200A+p7Vf2yfZfSYH9sHf9x4KBx0z4SeD/wd3V4b0qPsbldv45aT1CC47n09JrqvT8Kt/qd6a1/Vv37aeCqen+DulA4GnhoHfcB4OyJ3pch1/+AccMPqX//B6W78JOBX1F2K25BaR/7Tv3cvk45kHPrnv/fh9Lx5B3T8ZsZtXpnas2d/3DWxxtl98IFXdcxQV1bULoPPrYOH0LpDrmAsob793X8Bj3/cwCl587Tu65/kte0Wdc19FH7M7n3cQynUtqW5tXwvox6rAylY8VJ1K7IlJ5m2/f8f3QQHG+grGA8qA6/Gzim1nIeZYv1h5RzKEEJwVmUraXTgOf3PNds4AX1+/YI653ZNXf+A1pfb/VD3bHrOiao6zOUxlUo+0W/TdmkfU39Uj6xPrYdpRsuwBO6rnsKr2vUtjaOpRzkuF0dPpSyJvg0SvvGl2uALALO6fm/JcAb6emL38VrB+bUv0+knDds1zr8eWBBvf9u4Ls9/7MJ5eDSzSd4vrGtrU2sdzRqts1j+vxldtOIvDbHAO+pDZHPpWyJ/I6y5rKKcubV9wPfpBy0SI7A+Z+y/ipGQZQT0G0A/B1l3/RmlB/9icAjKO1i52bmtZQ1xDsj4s3139+VpV3jnr74w37ttR3vrjp4J6Ur93OjnDrkvyjdpqHsGr0jIhZHOWX5eZRu07f2PNdYV+S769/b7u/1jkrNMUK/OQ1IlIMZTwQ+RTmo7Jqex55DOVbiG5n5245KXO9FxEnAQZQjqI+gnOrlNZSDtd6cmddHxKaUxs7nUM5NdfxY77jazXIox2pMUv8CyoGhY/vXT6TsKrmRcoaB8zPzzihn9t0XeCpwQpbeetY7BTO9ZsPjfqj25lkBPDUzb4hykaA7RmntfZTVUFgE7Ek5Gv+MiHgRJSQ+k5kXRLneyDuA4zPzFx2WO9aLbVXP8GaUNeEzKKfXuSMiDqHU+z1KN+55dZpP9dZfv3tTPpbg/lDvqNbsKdnvh+qa68GUnkrPyMzbu65pfTTZ1kGW7pMfjoirgf0jYjnls3gI8LGI+C1l6++EcQuFTrY2cnW3zj2Ay+oKxwcp1wYZO7D0q1FO730xpXfPfMp5xCaqf9qP1xileke15qE2snmbWTdmaKP++nZj3InmWL3FvyFl1+ExwKZ13KMpWyAbDLPGtdT/AsoxP6dQjlfavo6/hHIczdh0+9fv1FbWez+ouesCvHX44dczanob6Hs61qMlKOcH+xCl88Rk0/03ymni9+n68xkLtXHjHk3Zvz6fciboGynHmzyQchnTC1iza/d21rt+1TzZzd5W92M5IicHHCVZTvMyL8uv/E7KgZWXTDRd/XsOpZH8/04wzVA+nyheSWm0JyI2j4gDIuJBmfkbynEGz6J0JT6Bsr/9oMz8GrARPedNysyrrHf9qHltbDCX+hA9lx4dG6Y0aN5OubDOdpQrQt7rpH5d95jqqWMO5YDRF1PaXo6gnPrlJuCLmXlmRHwVeGdmXh4RZ1C2ql5OeZ0re98D610/al4btzykdVCPkyFXn3zwlVGug/KAzNyDchqXh1N+/C+k59xBUU5Mt0ZwjD3fsNQ14bH+/3dRDkL7BeWaFAdl5l6UI5cPiogdKMcWvCoinklZbnwDuDkzr8/Mu2Karw8yavWOas0tZlQx0kwXEc+OiHOAPSNioyjXBP8xsAewPfD1iNgsS1/7s4FllAP/vhUR/xjlAMHM1Wc03SvK6fvbT0zXhyzujoj5UU67fwPljLzbUk6yCOUAtD9Qen59iLJv/n2Ui5l9LEuvsbHnm9YtqFGrd1RrbmFXXWkK6pbBhymnD/lnyr7pVZQrQ/4DJSi+yOprt9xC2SVxVWZ+LCLOB+7OzDvq882nnNRuQ+CNmXndEF7D+K2dt1N2qV0bEadk5mcj4qOUk+j9MDOvioh5lMb9X0XEqzLzlsme7/5e76jWvK4MD2lqtgLmZ+YusPpHHRFPo5yi/lbgzMx8cX18DqUB9Ba457Kq1MdeTDk77jG946dT70IoIo6gXBXy9sycH+WAxC9ExDeBL1GuSbOY1af6vqm+hlvq/8/Ocu2HoQTHKNQ7qjX3wwZzaQoiYivgC5TrUW9IuQDTAyjXfX475QJMP6/THknpYfUb4M89C4SxU4vMpezLnu6F2R6UKz0uyczbImJH4HnAX1MaYR8J7F4fO43SKPuGiHgTZb/814BTM/NevcWsd3RrHpicAf2FvXmb6TdKYCwCrgF+TtmF9W+UsxT/B+Wqhi+l7MP+AfConv+dNeRa5wMnU9pb9q7jHgn8kXJ5UoCnUK7h8so6vGV9HTtTToi5oLd+pvHMvaNW76jWPOibu62kKcjSVrEkIs6lBMjGmfn72nf/d5Rrvi8ATs7ML43736HteqhtKVcBr8nMV9RxG2TmdRHxEWD3OumVlBDcOSJ+lJlXRsSJwLzM/BnlWujDaNcYqXpHtebp4G4rqQ8RcSrl7Kb/NG78Gie6G3JNvwBekJm/iYijKAcqvgm4DvglcFRmfr92Dz0SuCkz39NFraNYL4xmzYNmeEgNakP4PMo5hl4CXE45hfrYWmRkxz+qKNemvwD4WR31vsy8oD52FKV95ll1+K+BqzPzmp42maG+hlGrd1RrHjTDQ2pUG0WPBk7LzB/WcTNqYRARnwC2zMyDJ3jsCmBxZp48/MomNmr1wmjWPEiGh9SHiOFcO6FVRGwCXE3Zv/7nWufrgXOBG3IIx5W0GLV6YTRrHiSPMJfWUW3ozJkWHHDPpUbfTrluyG6U4wl2An47tlCLGXS6i1GrF0az5kFyy0NaT9UF1y2UY06Oy8wfd1zSfRq1emE0ax4Uw0Naj0XEwzLz5p7hGd0tdNTqhdGseRAMD+l+oMuuw+ti1OqF0ay5H4aHJKnZetuYI0maPoaHJKmZ4SFJamZ4SJKaGR6SpGaGhySp2f8H2y7ensF6y+wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import scale\n",
    "logreg_new = LogisticRegression(C=1, solver='liblinear').fit(scale(rich_features_final), target)\n",
    "\n",
    "feature_names = rich_features_final.columns.values\n",
    "x = np.arange(len(feature_names))\n",
    "plt.bar(x, logreg_new.coef_.ravel())\n",
    "_ = plt.xticks(x + 0.5, feature_names, rotation=30)\n",
    "\n",
    "# Rich young women like Kate Winslet tend to survive the Titanic better\n",
    "# than poor men like Leonardo.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Non-linear models: ensembles of randomized trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sklearn` also implement non linear models that are known to perform very well for data-science projects where datasets have not too many features (e.g. less than 5000).\n",
    "\n",
    "In particular let us have a look at Random Forests and Gradient Boosted Trees:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random Forest CV scores:\n",
      "min: 0.787, mean: 0.806, max: 0.848\n",
      "Wall time: 2.76 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rf = RandomForestClassifier(n_estimators=100)\n",
    "scores = cross_val_score(rf, rich_features_final, target, cv=5, n_jobs=4,\n",
    "                         scoring='accuracy')\n",
    "print(\"Random Forest CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gradient Boosted Trees CV scores:\n",
      "min: 0.793, mean: 0.819, max: 0.859\n",
      "Wall time: 179 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "gb = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,\n",
    "                                subsample=.8, max_features=.5)\n",
    "scores = cross_val_score(gb, rich_features_final, target, cv=5, n_jobs=4,\n",
    "                         scoring='accuracy')\n",
    "print(\"Gradient Boosted Trees CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both models seem to do slightly better than the logistic regression model on this data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**:\n",
    "\n",
    "- Change the value of the learning_rate and other `GradientBoostingClassifier` parameter, can you get a better mean score?\n",
    "\n",
    "- Would treating the `PClass` variable as categorical improve the models performance?\n",
    "\n",
    "- Find out which predictor variables (features) are the most informative for those models.\n",
    "\n",
    "Hints:\n",
    "\n",
    "Fitted ensembles of trees have `feature_importances_` attribute that can be used similarly to the `coef_` attribute of linear models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%load solutions/04B_more_categorical_variables.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic Regression CV scores:\n",
      "min: 0.770, mean: 0.789, max: 0.816\n",
      "Gradient Boosting Trees CV scores:\n",
      "min: 0.792, mean: 0.825, max: 0.854\n"
     ]
    }
   ],
   "source": [
    "features = pd.concat([data.get(['Fare', 'Age']),\n",
    "                      pd.get_dummies(data.Sex, prefix='Sex'),\n",
    "                      pd.get_dummies(data.Pclass, prefix='Pclass'),\n",
    "                      pd.get_dummies(data.Embarked, prefix='Embarked')],\n",
    "                     axis=1)\n",
    "features = features.drop('Sex_male', 1)\n",
    "features = features.fillna(features.dropna().median())\n",
    "features.head(5)\n",
    "\n",
    "\n",
    "logreg = LogisticRegression(C=1, solver='liblinear')\n",
    "scores = cross_val_score(logreg, features, target, cv=5, scoring='accuracy')\n",
    "print(\"Logistic Regression CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))\n",
    "\n",
    "gb = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,\n",
    "                                subsample=.8, max_features=.5)\n",
    "scores = cross_val_score(gb, features, target, cv=5, n_jobs=4,\n",
    "                         scoring='accuracy')\n",
    "print(\"Gradient Boosting Trees CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%load solutions/04C_feature_importance.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEbCAYAAAA1T5h7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHcFJREFUeJzt3Xm8HHWZ7/HPNxsgILJEVIieCFHJIKhExGVcEAQvAyiCggsuSBTNuOBF44aK15HRUdy4DhE3VETQcUSIA86oo1xECctFkcsYECUXL0RRHAcQkjz3j+fXPyrNSdLnnF4qyff9evXrdFVXVz2nu7qe+i31K0UEZmZmANNGHYCZmbWHk4KZmVVOCmZmVjkpmJlZ5aRgZmaVk4KZmVVOCmZmVjkpmJlZ5aRgZmbVjFFteKeddoqxsbFRbd7MbKN0xRVX/C4iZg9q/SNLCmNjYyxbtmxUmzcz2yhJ+vUg1+/qIzMzq5wUzMysclIwM7PKScHMzConBTMzq5wUzMysclIwM7PKScHMzConBTMzq0Z2RbP119jiC4e6vZtOPWSo2zOz4XBJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrekoKkg6WdL2k5ZIWr2e5IyWFpAX9C9HMzIZlg0lB0nTgdOC5wHzgGEnzx1luW+ANwE/6HaSZmQ1HLyWFfYHlEXFjRNwDnAMcPs5y7wc+BNzdx/jMzGyIekkKuwA3N6ZXlHmVpMcDcyLigj7GZmZmQ9ZLUtA486K+KE0DTgPessEVSQslLZO0bOXKlb1HaWZmQ9FLUlgBzGlM7wrc0pjeFtgT+IGkm4D9gPPHa2yOiCURsSAiFsyePXvyUZuZ2UD0khQuB+ZJmitpFnA0cH7nxYi4IyJ2ioixiBgDLgMOi4hlA4nYzMwGZoNJISJWAYuAi4DrgHMj4lpJp0g6bNABmpnZ8PR0O86IWAos7Zp38jqWfebUwzIzs1HwFc1mZlY5KZiZWeWkYGZmlZOCmZlVTgpmZlY5KZiZWeWkYGZmlZOCmZlVTgpmZlY5KZiZWeWkYGZmlZOCmZlVTgpmZlY5KZiZWeWkYGZmlZOCmZlVTgpmZlY5KZiZWeWkYGZmlZOCmZlVTgpmZlY5KZiZWeWkYGZmlZOCmZlVTgpmZlY5KZiZWeWkYGZmlZOCmZlVTgpmZlY5KZiZWeWkYGZmlZOCmZlVTgpmZlY5KZiZWeWkYGZmlZOCmZlVTgpmZlY5KZiZWeWkYGZmVU9JQdLBkq6XtFzS4nFef62kn0m6WtIlkub3P1QzMxu0DSYFSdOB04HnAvOBY8Y56J8dEY+NiMcBHwI+2vdIzcxs4HopKewLLI+IGyPiHuAc4PDmAhHxp8bk1kD0L0QzMxuWGT0sswtwc2N6BfCk7oUkvR44EZgF7N+X6MzMbKh6KSlonHn3KwlExOkRsRvwNuBd465IWihpmaRlK1eunFikZmY2cL0khRXAnMb0rsAt61n+HOB5470QEUsiYkFELJg9e3bvUZqZ2VD0khQuB+ZJmitpFnA0cH5zAUnzGpOHAL/sX4hmZjYsG2xTiIhVkhYBFwHTgc9FxLWSTgGWRcT5wCJJBwD3An8AXj7IoM3MbDB6aWgmIpYCS7vmndx4/sY+x2VmZiPgK5rNzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMqp6SgqSDJV0vabmkxeO8fqKkX0i6RtK/SXpE/0M1M7NBm7GhBSRNB04HDgRWAJdLOj8iftFY7CpgQUTcKekE4EPAiwYRMMDY4gsHtepx3XTqIUPdnpnZqPRSUtgXWB4RN0bEPcA5wOHNBSLi+xFxZ5m8DNi1v2Gamdkw9JIUdgFubkyvKPPW5TjgO1MJyszMRmOD1UeAxpkX4y4ovRRYADxjHa8vBBYCPPzhD+8xRDMzG5ZeSgorgDmN6V2BW7oXknQA8E7gsIj4y3grioglEbEgIhbMnj17MvGamdkA9ZIULgfmSZoraRZwNHB+cwFJjwfOIBPCbf0P08zMhmGDSSEiVgGLgIuA64BzI+JaSadIOqws9mFgG+A8SVdLOn8dqzMzsxbrpU2BiFgKLO2ad3Lj+QF9jsvMzEbAVzSbmVnlpGBmZpWTgpmZVU4KZmZWOSmYmVnlpGBmZpWTgpmZVU4KZmZWOSmYmVnlpGBmZpWTgpmZVU4KZmZWOSmYmVnV0yiptm5jiy8c6vZuOvWQoW7PzDYvLimYmVnlpGBmZpWTgpmZVU4KZmZWOSmYmVnlpGBmZpWTgpmZVU4KZmZWOSmYmVnlpGBmZpWTgpmZVU4KZmZWOSmYmVnlpGBmZpWTgpmZVU4KZmZWOSmYmVnlpGBmZpVvx2k2BL5tq20sXFIwM7PKScHMzConBTMzq5wUzMysclIwM7PKScHMzKqekoKkgyVdL2m5pMXjvP50SVdKWiXpyP6HaWZmw7DBpCBpOnA68FxgPnCMpPldi/0GeAVwdr8DNDOz4enl4rV9geURcSOApHOAw4FfdBaIiJvKa2sGEKOZmQ1JL9VHuwA3N6ZXlHlmZraJ6SUpaJx5MZmNSVooaZmkZStXrpzMKszMbIB6SQorgDmN6V2BWyazsYhYEhELImLB7NmzJ7MKMzMboF6SwuXAPElzJc0CjgbOH2xYZmY2ChtMChGxClgEXARcB5wbEddKOkXSYQCSnihpBXAUcIakawcZtJmZDUZPQ2dHxFJgade8kxvPLyerlczMbCPmK5rNzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMKicFMzOrnBTMzKxyUjAzs8pJwczMqhmjDsA2PWOLLxzq9m469ZChbs9sU+aSgpmZVU4KZmZWOSmYmVnlpGBmZpWTgpmZVU4KZmZWuUuqmW323I36Pi4pmJlZ5aRgZmaVq49skzbMaoE2Vwk0+TOx9XFJwczMqp5KCpIOBj4OTAfOjIhTu17fAjgL2Af4PfCiiLipv6Ga2abEjbvttMGSgqTpwOnAc4H5wDGS5nctdhzwh4jYHTgN+Pt+B2pmZoPXS/XRvsDyiLgxIu4BzgEO71rmcOCL5fnXgWdLUv/CNDOzYeglKewC3NyYXlHmjbtMRKwC7gB27EeAZmY2PIqI9S8gHQUcFBGvLtMvA/aNiL9tLHNtWWZFmb6hLPP7rnUtBBaWyUcD1/frHxminYDfjTqIwrHcX1vigPbE0pY4wLGMZ6JxPCIiZg8qmF4amlcAcxrTuwK3rGOZFZJmANsBt3evKCKWAEsmF2o7SFoWEQtGHQc4ljbHAe2JpS1xgGNpcxwdvVQfXQ7MkzRX0izgaOD8rmXOB15enh8JfC82VAQxM7PW2WBJISJWSVoEXER2Sf1cRFwr6RRgWUScD3wW+JKk5WQJ4ehBBm1mZoPR03UKEbEUWNo17+TG87uBo/obWmu1qfrLsdxfW+KA9sTSljjAsYynLXEAPTQ0m5nZ5sPDXJiZWeWkYGZmlZNCH4z66u0yFMnI4zCzjZ+TwhRImgbQ7H47zANzZ1sRsbrM2nIUcfRbJ8mNaNsj/U1I2m7UMXRI2qlcd9R6kraU9BZJz29BHM+XtOco4yixbCHpCeV5z8eDVux8G6uIWAMg6TBJH5b0OOCBQ9x+lO0fIelHwHskvan52sakkWRXl+mxMgLvILe5jaRPS3rHILfTYxxnAF8FPinpwSOO5VPAecBpknYbVSy9KCMl/BR4DDAmabsRxXECsIzsiXmupA9KevKIYjmevMbsVZJmTOR44KQwQc2zOEkzJX0ceC1wHfBu7ruIb+DbL9P7AseSw4dcAbxJ0lMGGUO/SXoirJVkj5L0C+CtwFmDOnOWtANwNrAb8BJJj4mINcMuqUjaHfgecCd5jc8uwHGSpg+7xCdpLnkwuZccGXn3ElMrS5+SHgkcCrw8Io6PiNMi4o4RxPEY4DDgxRHxYjIx3AZ8cASxfIS8iPioiFgErJnI+50UJqgcNLaVdEhE3Juz4r+RQ3vsAfyfQWy3c6Aq258paa9ysNyHPKA8E3gH8HcRcekgYhiEchZ6aqfYL2ke8DfAgcCHyB9X96i8fRERtwMfIw9+ZwPvL/NXr+99A/A74L0R8eaI+BM59PyxEbF62CW+iPgVcHiJ5W7gUuAv5bU2lj6fDkyPiKskTRth1dtTgVURcY0kRcS15EW9d0h66zACUJoJCHhnRFwvadtyzJjV63qcFCZI0iHAZ4CDJO0EPEnSr4CHA0+JiIslPaTfZ1WNKpWXAecCHwC2BZYDHwHmAftFxBJJsyX9VT+332+Nz+cW8oB8dKm/fjA5ltY7gK8Bx0XENwe4/e+Xz/ZMYE75fhlyXfqfgX8t250J/Bb4uaStRnF2HhH/UUopXwDeRO7jH+6U6FpmNXCJpOkRsaZT2hxRHD9uVtWUBP95YPeJHJQnq5G0HwU8XdKHyOq/84CLJT26l/U4KaxDybrdVTV7AKcAt0XEG8gi9o+Br5Uzqz9KOhJ4NbD1FLf/rFKU70w/SNJXgIPIg+VjybPoX5IH1Vsj4i+SDiDvabHHVLY/KI0ST+eHcxfwb8DdwIvJhPBi8uz5KRHxeUmPlPTM8v6+HCQb2+/8vRU4A3h7mV41rANyRKwq9yqhlD4fC6yJiLtGcXZeznRXA6dFxI7Ay8iqrf1L0mqTv5Cl5O2h/m6nS9pR0l935g0hjjtLHA8q2+wcO3YCpnW+30GSNK3sPx8AZgMLgGuAC4BfAa+T9IANrcdJYR0irSkHpIMkzY6I68gDbqcO/A7gO+SZwJdLRv7vwJUR8efJbrtR1/1FScc1XtoZeFNEnAOcXOLYntwJnifpG2TS+mREfH2y2x+kRonnCEknlHr8m4BvAi8CVgIXAw8AniipkwQPKAerCR0ke61OKAeOc4BbJL1G0oHA0yayrX7EURxI/pCRdHg5GembDcXS+IyvKdN3k2OaPbgcdIamVNV+WNKCMr1W7BFxLhBkg+oO5Xe7Gngc8FJJW/UjsfYYB8ArSxydEssaYGk/q7XWFUtnmxHxY+BdEbE/8OmI+CJ5jNiH/KzWy0mhKGcXH5D01DItSX9PHqz2Af6lVBd9BrhB2dOAiPgucALZa+SiiNivjBU1FauBK8s6j5N0LJkQfgHsXbZ7FvAE4BiyCukpwNsi4imdhDCKqocNkbSzpAuAV5JnV0sk7QV8G7gRWERWWfy2/D0ReEdEvGsyCaHReL2b1tOAXA4md5G9Rz5Nti/8ZsL/4BTiaFRZTQP2lnQhuW/9sR9xTCQWWKt325OA55H739A09t+7yFv+0jjYNrsunwz8FXCepEMkfY68p/yF5TsdZhyPJXsePUfSZ8lqybcD75lqHF2x3DleLB0RsbpUqd2r7Mn2duAqekgKRMRm/yCre75PniluV+btTvYm2oI8c7sb+Ep57QhyuPCHDSCWznhUXwJOIksDnyZ3qiXAa8ibbAD8I/AVYF7XOqaP+jNt/i9d8/YADi3POwfei8r0k8jReJ9cpnfoeu+0ScTwUOAs4Cedz209yx4P/IzsQdLvz2IicfyW7M12+IC+l55iIU9EPg5cDbxkhPvRMWW/OGBd+wFZsjyOPCP+4CB+Az3GsTXZE/AfSiwPJG9p3Nd4eozlccB7yZuZ/V2vMYzkS27Tg2zYXAPs2Zi3bfm7TflRfIc8K/8zWc8NeU/qhQOIp5MUjiDPkCHPmG8Efgj8D7Jq4QdkA/MlZHe8kX+Wjf9hWtf0ieWH8qgyvQOZVP8nWf11NfCi8trJnQNQ47PobWfuWo7shfFPnc+xh8985/WtbwL//6TiKMvOAvbvRxx9iGUGsGB9322f9xt1Te8H/Jw8KbqUrFKdPt6y461jCt9fP+IY7yA97rKDioVMUC+kkfh7+f48Siog6UzybPU8SWeRjbYnSdqe7LL4zohYIelS4KERMVfSThExsFv5KXsZHUomrL3IborPI+t2bwKWRsQVkr4IfDYifjioWHrVKdpGZw+VHkj23X4F2SD/dOD5wFbA+4DjIyIkfReYT/bgUuR9vqcSxzFkUf9fyDO2G4E/AA8BbiWrFW5tLD+thN2Je8ZUY+hTHDOjT3X4bflM1hPf9Gh0BS7bfzv5WzxT0v7AS4ErIuL07valzvubVabN14cdR3m+1u9hmLE0qwk76yI7L7hNoUdvBM6WdA1wLbC4zJ9DDh3xLElvA75L9lBhkAmhuICstloZEfMjG4veDnyBLC3sK+kqcgcZeUKAWi8fknaR9FWyiuLJwHMi4t3kNRyLyaq4R5O9Ic4h79Xxwsh++atgcu0hknaV9EOyveLPkQ2kPyKT6nbAjsAhQO2V0vnxlLjnlf9jqklpqnE8qvzAp5wQ+hjLwBIC1DrwbSQtlLRrOaDtAXR64F1OlpQPkrRziW2a7uvNtlrSlsDMzn44yjgkzZpKHFONJe5rN9pS0pYxkWteNlSU2Fwe5I/mG43pTinqBcCXyV5Hs4cYj4DTgAPLdHc1wKHAw1vwuU3r+rzeQLZzHEu2ifwIeGJ5bR7Z2DVGHpROA949ye3er2qA7Mr6lsb0A7rfA3yLTFLN+dsDnyRLMw/cGONoWyy97jddcf6MLJmfBryF7DxxAaVaryxzDXmBZvf6FpEnbXM3xjjaFEtfv+iN+UGWmlYAe5XpWY3XthhBPCLr3Q9l7XrSCddLDvDzasbVSQ7/CNxYns8k2w0WAduXeR+kNCx3/78T+Wwaz48F9i7PX06ePX2j/Ih+Q3Zz3bn8oK4jS1nTG+9fTNbNHjSZ76gNcbQtlg3EuVXX9IPK3/eTjcV7lwPh+8j2vk+SVV6Hkm0hLwV2bbz/YPLCv3dP5HfaljjaFkuEk0L3l/Nk4LJRx9GIZ4dRxzBOTE9rxgU8mywZvJWsbnsY2XWx0yvib8jL/Z9dpreh0VuKTH4bTAhksfmVnR8QsD851tOXgI8CJzXieQZ50dCryVLeFuWHM7drnc8tP5yZE/j/WxFH22LpMd43kr1gtinT7yU7UYhMQv9OdqA4qrw+kzz5eD3ZoHpEY13Tua8U/5CNMY62xVLX0+8vfmN/lC9ir1HH0RVTW0oHbyG7Sz6yTB9NnpHsQ7YffI1MDAuB7zXetwR4M7DlZP+ncgA7oxz4RJY+HkueSf0vsm71mMbyu5cfx2k0iuXlBzXpHjRtiaNtsWwgzhnl757kOF2dLsdnAXuU5+8FLm68Z2vgb4Gdxllfp1S69cYYR9ti6X64ofn+/joirhl1EE1Rvu1RUo7dMpO8Yntv5VXXW5MHpYeQ11P8MCJuJg8890o6qbz9PZGjV97dWV8v/1PXhVXfAv4v2SC6HXA6We99GfDP5FnTwZK2Vw418nXyyvI3R6MXRkxifJy2xNG2WHqMt9lIfS95oeWhyiEo7iL70ENexX6PpFOVF4ZeCjwCuKOxru4rd/9rY4ujbbGMa1BnB35seg+yGuh2cpz9B5GD1v2aTAIPLctsS1ZNHAp8sMzrNEL3dDYK9+tv/VKyx8UC4BNkTyXIOtZjyvPXkj3HFpF9/bdqvH9SZ8FtiaNtsUwi9j2Ad5LX/OxOjnX1fbJk+XRKVRXZEeENZA+7J2yqcbQtlu6HSwrWE0nbkm0FPwHOjog/AjeQ1UefiojfKu/jcC5Z/P12RHQGl4vyd71no5IeXbpDRpneW9JPyQv5toyIZWQS2kd5I5UA3qgcLO855A/smxFxT0Tc1egmONGSQSviaFssPcY7vWt6B7Iq65FkG8dysjPCjmTHjleRQ8icQVZ9fCIiXhERV3a6x27McbQtlp4MI/P4sfE82MAZJNmQdSZZd70VOezGZWQyuAZ49UTWV5YZI6s6ribbH44o898NnNC17J5kF72Xlen3kT0xjurD/96KONoWyyTjfxb3dZt8G1n10ezhdB7ZM2rbsi89f6L7zcYUR9tiWW+co9pp/Gj3g0ajcJnuVAHNIi+xfxP3DQeyG3lWOuEeK+Xgt4YcnnlLckiMz5MN1mdS+tizdtXHK8kGub3HWd+kGuXbEkfbYplE7C8gh2H4InmiMK/M/zmNsZzIC+YuBXbZlONoWyw9xTvKjfvRjgeNC9DIhsoPkw3u61puf7Lf+8HjLDPh8WbIs+FO74vHkddn7En2tHgr5aLBEt988kzqhazdm2nKZ1FtiaNtsawjvvEGO9yNbOweI7u/3kZep/IA8vaQl9E4caD0YtsU4mhbLFN5uE3BiLxvxJzIvfJe8kY+Px9vufL3e+SVyv8xzjKTuZXl88izKMiSyBZke8WV5HUPn5L0NHJgwtcDd0XEubF2b6Z+1JG3JY62xVKVOu1XAa8r0ztJOkzSNhFxA9nv/plkW9MnyHry50cO574F2fjdie/GjT2OtsXSDx4QbzOkrsHNlOP4f5e8i9UlZAPYq8n9Y3XXe6cN6GBzJnkJ/0qy4fo8SSLvIHUCOVbSxRHxhX5vu41xtC2WRkwzyDv+vYQsvRxP3u/hd8CXI+JC5c2mTo6I6yR9kyx9voLcv1ZGfwYbbEUcbYulH5wUNiNlYKy7G9OvIi9GuyQi/lPSE8i66deTY6pcEhEryrJr9Yceb31TjG0r4E/k/SzuHG97jWUHkpjaFEebYimJSHHfIGs7k9VYh5D3xlgh6bVk99iPksNk3EZeK3ES2dXysxHxn1OJtS1xtC2WfnP10WZA0rMlfY+8peUWyluMXkL2hpgH/JPyFoJXkjfuWEZekPZtSf+gvHAtGj+AA5VDjD+0XzFG3iFrEdmLBrI+fK0fyfoOiptaHG2KJdIaSWOS5kYOsb2UvE5i97LYxeTZ8WPINqndyJvMfDwiPtY5+E0l1rbE0bZY+q6fDRR+tOtB9lw5nWzMOrZMzyAvLDuYbKT8Cjm2Smfoiv2Ac8rzJ1FGOC3TY+S4Ol9jMHedm0b+iHYb8efWijhGGQv3H7HzXeR9PH5E6XZM3g3wzMYyXwOOLM+nfNe8NsXRtlgG+ejcE9Y2TbsAYxGxH9xXRJW0DzlS5h3kzVVeUl6fQTaI3Q4QET/prEjSSyj3T27O76cS26Mi4jZp7ZuXDFNb4hhVLM2qDEnHk0Np/CUixpQXKH5J0rfIe4ifLelUcojt+WQ9OhFxe3n/9Mix/CdTXdSKONoWy6C5TWETJmkX8sz+FLIHy6PJC85uJc9yXhgRV5VlTyB7HN0A3N3YgRURIWk28Pu27sg2NZKeRXZ9XRIR/yVpL7IH1DPIxtCHAU8tr51NNo6+UdKJZKP318l7mN+v19rGGEfbYhmqURdV/Bjcg0wEC8lhEK4i7+n8feAz5JnOBWS10sXk2CuPaLy3lUVbP/q+j4yRF8Yto9w/gTzY/Qn4WJl+PDlcxqvK9EPL/rMv2RNqj+Z+wyQulmtLHG2LZRQPVx9twiLiHmCJ8naMvyYvbPpD6XX0/4BfkgNzfT4ivtr1XpcINnGSxsh7Nb8mIl5Z5s2MiFskfRR4aln0l+RJxb6SfhQRv1SOyzMnIn5KdpmdSq+iVsTRtlhGxdVHmyFJXwF+HBGf6pq/1o3CbdMn6WrgBRFxg6TXkRcungjcAvxv4HUR8a+S5pNVIr+LiPdtqnG0LZZRcFLYDJQG5DlkH+qXkbdgPCkiOmczI21MtdGRNJfsnfbTMusDEXFZee11ZLvTM8v0M4CbIuLXjbamvuw7bYmjbbGMgpPCZqI0ki0ih73+QZm3Ue+81h+SPkneD+PIcV67Hjg1Ij6/ucTRtliGzUlhMyStfTWmbd4kbU32t58TEXeX/eMN5O08b42IWzanONoWy7D5iubNTGn4CicE64i8heO7gI+UPvc/Jrti/qZz8NOgb+zSojjaFsuwuaRgZp0D3O3ktSqLI+KSzTmOtsUyTE4KZgaApB0j4veN6ZF0p2xLHG2LZVicFMxsLW3pmtyWOKBdsQyak4KZmVWbZEOJmZlNjpOCmZlVTgpmZlY5KZiZWeWkYGZmlZOCmZlV/x9Jz7rX5y75KwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "gb_new = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,\n",
    "                                    subsample=.8, max_features=.5)\n",
    "gb_new.fit(features, target)\n",
    "feature_names = features.columns.values\n",
    "x = np.arange(len(feature_names))\n",
    "plt.bar(x, gb_new.feature_importances_)\n",
    "_ = plt.xticks(x + 0.5, feature_names, rotation=30)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "GradientBoostingClassifier?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated parameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of changing the value of the learning rate manually and re-running the cross-validation, we can find the best values for the parameters automatically (assuming we are ready to wait):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 9.48 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "             estimator=GradientBoostingClassifier(criterion='friedman_mse',\n",
       "                                                  init=None, learning_rate=0.1,\n",
       "                                                  loss='deviance', max_depth=3,\n",
       "                                                  max_features=None,\n",
       "                                                  max_leaf_nodes=None,\n",
       "                                                  min_impurity_decrease=0.0,\n",
       "                                                  min_impurity_split=None,\n",
       "                                                  min_samples_leaf=1,\n",
       "                                                  min_samples_split=2,\n",
       "                                                  min_weight_fraction_leaf=0.0,\n",
       "                                                  n_estimators=100,\n",
       "                                                  n_iter_no_change=None,\n",
       "                                                  presort='auto',\n",
       "                                                  random_state=None,\n",
       "                                                  subsample=0.8, tol=0.0001,\n",
       "                                                  validation_fraction=0.1,\n",
       "                                                  verbose=0, warm_start=False),\n",
       "             iid='warn', n_jobs=4,\n",
       "             param_grid={'learning_rate': [0.05, 0.1, 0.5],\n",
       "                         'max_depth': [3, 4, 5], 'max_features': [0.5, 1]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring='roc_auc', verbose=0)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "gb = GradientBoostingClassifier(n_estimators=100, subsample=.8)\n",
    "\n",
    "params = {\n",
    "    'learning_rate': [0.05, 0.1, 0.5],\n",
    "    'max_features': [0.5, 1],\n",
    "    'max_depth': [3, 4, 5],\n",
    "}\n",
    "gs = GridSearchCV(gb, params, cv=5, scoring='roc_auc', n_jobs=4)\n",
    "gs.fit(rich_features_final, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us sort the models by mean test score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.869368</td>\n",
       "      <td>0.026307</td>\n",
       "      <td>{'learning_rate': 0.05, 'max_depth': 3, 'max_features': 0.5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.861091</td>\n",
       "      <td>0.019516</td>\n",
       "      <td>{'learning_rate': 0.05, 'max_depth': 3, 'max_features': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.871461</td>\n",
       "      <td>0.025538</td>\n",
       "      <td>{'learning_rate': 0.05, 'max_depth': 4, 'max_features': 0.5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.869763</td>\n",
       "      <td>0.021538</td>\n",
       "      <td>{'learning_rate': 0.05, 'max_depth': 4, 'max_features': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.869849</td>\n",
       "      <td>0.028569</td>\n",
       "      <td>{'learning_rate': 0.05, 'max_depth': 5, 'max_features': 0.5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.867352</td>\n",
       "      <td>0.025302</td>\n",
       "      <td>{'learning_rate': 0.05, 'max_depth': 5, 'max_features': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.871358</td>\n",
       "      <td>0.028221</td>\n",
       "      <td>{'learning_rate': 0.1, 'max_depth': 3, 'max_features': 0.5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.864049</td>\n",
       "      <td>0.018794</td>\n",
       "      <td>{'learning_rate': 0.1, 'max_depth': 3, 'max_features': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.875952</td>\n",
       "      <td>0.027863</td>\n",
       "      <td>{'learning_rate': 0.1, 'max_depth': 4, 'max_features': 0.5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.872597</td>\n",
       "      <td>0.024160</td>\n",
       "      <td>{'learning_rate': 0.1, 'max_depth': 4, 'max_features': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.867399</td>\n",
       "      <td>0.024426</td>\n",
       "      <td>{'learning_rate': 0.1, 'max_depth': 5, 'max_features': 0.5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.859261</td>\n",
       "      <td>0.020529</td>\n",
       "      <td>{'learning_rate': 0.1, 'max_depth': 5, 'max_features': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.865505</td>\n",
       "      <td>0.024277</td>\n",
       "      <td>{'learning_rate': 0.5, 'max_depth': 3, 'max_features': 0.5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.869727</td>\n",
       "      <td>0.031167</td>\n",
       "      <td>{'learning_rate': 0.5, 'max_depth': 3, 'max_features': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.863554</td>\n",
       "      <td>0.029018</td>\n",
       "      <td>{'learning_rate': 0.5, 'max_depth': 4, 'max_features': 0.5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.858756</td>\n",
       "      <td>0.036530</td>\n",
       "      <td>{'learning_rate': 0.5, 'max_depth': 4, 'max_features': 1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.853316</td>\n",
       "      <td>0.026911</td>\n",
       "      <td>{'learning_rate': 0.5, 'max_depth': 5, 'max_features': 0.5}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.846714</td>\n",
       "      <td>0.017781</td>\n",
       "      <td>{'learning_rate': 0.5, 'max_depth': 5, 'max_features': 1}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_test_score  std_test_score  \\\n",
       "0   0.869368         0.026307         \n",
       "1   0.861091         0.019516         \n",
       "2   0.871461         0.025538         \n",
       "3   0.869763         0.021538         \n",
       "4   0.869849         0.028569         \n",
       "5   0.867352         0.025302         \n",
       "6   0.871358         0.028221         \n",
       "7   0.864049         0.018794         \n",
       "8   0.875952         0.027863         \n",
       "9   0.872597         0.024160         \n",
       "10  0.867399         0.024426         \n",
       "11  0.859261         0.020529         \n",
       "12  0.865505         0.024277         \n",
       "13  0.869727         0.031167         \n",
       "14  0.863554         0.029018         \n",
       "15  0.858756         0.036530         \n",
       "16  0.853316         0.026911         \n",
       "17  0.846714         0.017781         \n",
       "\n",
       "                                                          params  \n",
       "0   {'learning_rate': 0.05, 'max_depth': 3, 'max_features': 0.5}  \n",
       "1   {'learning_rate': 0.05, 'max_depth': 3, 'max_features': 1}    \n",
       "2   {'learning_rate': 0.05, 'max_depth': 4, 'max_features': 0.5}  \n",
       "3   {'learning_rate': 0.05, 'max_depth': 4, 'max_features': 1}    \n",
       "4   {'learning_rate': 0.05, 'max_depth': 5, 'max_features': 0.5}  \n",
       "5   {'learning_rate': 0.05, 'max_depth': 5, 'max_features': 1}    \n",
       "6   {'learning_rate': 0.1, 'max_depth': 3, 'max_features': 0.5}   \n",
       "7   {'learning_rate': 0.1, 'max_depth': 3, 'max_features': 1}     \n",
       "8   {'learning_rate': 0.1, 'max_depth': 4, 'max_features': 0.5}   \n",
       "9   {'learning_rate': 0.1, 'max_depth': 4, 'max_features': 1}     \n",
       "10  {'learning_rate': 0.1, 'max_depth': 5, 'max_features': 0.5}   \n",
       "11  {'learning_rate': 0.1, 'max_depth': 5, 'max_features': 1}     \n",
       "12  {'learning_rate': 0.5, 'max_depth': 3, 'max_features': 0.5}   \n",
       "13  {'learning_rate': 0.5, 'max_depth': 3, 'max_features': 1}     \n",
       "14  {'learning_rate': 0.5, 'max_depth': 4, 'max_features': 0.5}   \n",
       "15  {'learning_rate': 0.5, 'max_depth': 4, 'max_features': 1}     \n",
       "16  {'learning_rate': 0.5, 'max_depth': 5, 'max_features': 0.5}   \n",
       "17  {'learning_rate': 0.5, 'max_depth': 5, 'max_features': 1}     "
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make sure the entire params column will display\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "#Zip gs.cv_results_ into a table \n",
    "cv_results = zip(gs.cv_results_['mean_test_score'], gs.cv_results_['std_test_score'], gs.cv_results_['params'])\n",
    "#Convert table into a dataframe\n",
    "df = pd.DataFrame(cv_results, columns = ['mean_test_score', 'std_test_score','params'])\n",
    "#Sort by mean test score and display\n",
    "df.sort_values(by=['mean_test_score'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8759521612869435"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'learning_rate': 0.1, 'max_depth': 4, 'max_features': 0.5}"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should note that the mean scores are very close to one another and almost always within one standard deviation of one another. This means that all those parameters are quite reasonable. The only parameter of importance seems to be the `learning_rate`: 0.5 seems to be a bit too high."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Avoiding data snooping with pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When doing imputation in pandas, prior to computing the train test split we use data from the test to improve the accuracy of the median value that we impute on the training set. This is actually cheating. To avoid this we should compute the median of the features on the training fold and use that median value to do the imputation both on the training and validation fold for a given CV split.\n",
    "\n",
    "To do this we can prepare the features as previously but without the imputation: we just replace missing values by the -1 marker value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Pclass_1</th>\n",
       "      <th>Pclass_2</th>\n",
       "      <th>Pclass_3</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.2500</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>71.2833</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.9250</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53.1000</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.0500</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare   Age  Sex_female  Pclass_1  Pclass_2  Pclass_3  Embarked_C  \\\n",
       "0  7.2500   22.0  0           0         0         1         0            \n",
       "1  71.2833  38.0  1           1         0         0         1            \n",
       "2  7.9250   26.0  1           0         0         1         0            \n",
       "3  53.1000  35.0  1           1         0         0         0            \n",
       "4  8.0500   35.0  0           0         0         1         0            \n",
       "\n",
       "   Embarked_Q  Embarked_S  \n",
       "0  0           1           \n",
       "1  0           0           \n",
       "2  0           1           \n",
       "3  0           1           \n",
       "4  0           1           "
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = pd.concat([data.get(['Fare', 'Age']),\n",
    "                      pd.get_dummies(data.Sex, prefix='Sex'),\n",
    "                      pd.get_dummies(data.Pclass, prefix='Pclass'),\n",
    "                      pd.get_dummies(data.Embarked, prefix='Embarked')],\n",
    "                     axis=1)\n",
    "features = features.drop('Sex_male', 1)\n",
    "\n",
    "# Because of the following bug we cannot use NaN as the missing\n",
    "# value marker, use a negative value as marker instead:\n",
    "# https://github.com/scikit-learn/scikit-learn/issues/3044\n",
    "features = features.fillna(-1)\n",
    "features.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the `Imputer` transformer of scikit-learn to find the median value on the training set and apply it on missing values of both the training set and the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(features.values, target, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SimpleImputer(add_indicator=False, copy=True, fill_value=None,\n",
       "              missing_values=-1, strategy='median', verbose=0)"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "imputer = SimpleImputer(strategy='median', missing_values=-1)\n",
    "\n",
    "imputer.fit(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The median age computed on the training set is stored in the `statistics_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([14.5, 29. ,  0. ,  0. ,  0. ,  1. ,  0. ,  0. ,  1. ])"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imputer.statistics_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imputation can now happen by calling  the transform method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_imputed = imputer.transform(X_train)\n",
    "X_test_imputed = imputer.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.any(X_train == -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.any(X_train_imputed == -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.any(X_test == -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.any(X_test_imputed == -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use a pipeline that wraps an imputer transformer and the classifier itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7988826815642458 0.8283580284610149 0.8587570621468926\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "imputer = SimpleImputer(strategy='median', missing_values=-1)\n",
    "\n",
    "classifier = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,\n",
    "                                        subsample=.8, max_features=.5)\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('imp', imputer),\n",
    "    ('clf', classifier),\n",
    "])\n",
    "\n",
    "scores = cross_val_score(pipeline, features.values, target, cv=5, n_jobs=4,\n",
    "                         scoring='accuracy', )\n",
    "print(scores.min(), scores.mean(), scores.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mean cross-validation is slightly lower than we used the imputation on the whole data as we did earlier although not by much. This means that in this case the data-snooping was not really helping the model cheat by much.\n",
    "\n",
    "Let us re-run the grid search, this time on the pipeline. Note that thanks to the pipeline structure we can optimize the interaction of the imputation method with the parameters of the downstream classifier without cheating:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1.67 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "             estimator=Pipeline(memory=None,\n",
       "                                steps=[('imp',\n",
       "                                        SimpleImputer(add_indicator=False,\n",
       "                                                      copy=True,\n",
       "                                                      fill_value=None,\n",
       "                                                      missing_values=-1,\n",
       "                                                      strategy='median',\n",
       "                                                      verbose=0)),\n",
       "                                       ('clf',\n",
       "                                        GradientBoostingClassifier(criterion='friedman_mse',\n",
       "                                                                   init=None,\n",
       "                                                                   learning_rate=0.1,\n",
       "                                                                   loss='deviance',\n",
       "                                                                   max_depth=3,\n",
       "                                                                   max_features=0.5,\n",
       "                                                                   max_leaf_nodes=...\n",
       "                                                                   n_iter_no_change=None,\n",
       "                                                                   presort='auto',\n",
       "                                                                   random_state=None,\n",
       "                                                                   subsample=0.8,\n",
       "                                                                   tol=0.0001,\n",
       "                                                                   validation_fraction=0.1,\n",
       "                                                                   verbose=0,\n",
       "                                                                   warm_start=False))],\n",
       "                                verbose=False),\n",
       "             iid='warn', n_jobs=4,\n",
       "             param_grid={'clf__max_depth': [3, 4, 5],\n",
       "                         'clf__max_features': [0.5, 1],\n",
       "                         'imp__strategy': ['mean', 'median']},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring='roc_auc', verbose=0)"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "params = {\n",
    "    'imp__strategy': ['mean', 'median'],\n",
    "    'clf__max_features': [0.5, 1],\n",
    "    'clf__max_depth': [3, 4, 5],\n",
    "}\n",
    "gs = GridSearchCV(pipeline, params, cv=5, scoring='roc_auc', n_jobs=4)\n",
    "gs.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>params</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.870452</td>\n",
       "      <td>0.016936</td>\n",
       "      <td>{'clf__max_depth': 3, 'clf__max_features': 0.5, 'imp__strategy': 'mean'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.869132</td>\n",
       "      <td>0.032918</td>\n",
       "      <td>{'clf__max_depth': 3, 'clf__max_features': 0.5, 'imp__strategy': 'median'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.855942</td>\n",
       "      <td>0.028338</td>\n",
       "      <td>{'clf__max_depth': 3, 'clf__max_features': 1, 'imp__strategy': 'mean'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.856940</td>\n",
       "      <td>0.025588</td>\n",
       "      <td>{'clf__max_depth': 3, 'clf__max_features': 1, 'imp__strategy': 'median'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.864322</td>\n",
       "      <td>0.031108</td>\n",
       "      <td>{'clf__max_depth': 4, 'clf__max_features': 0.5, 'imp__strategy': 'mean'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.866819</td>\n",
       "      <td>0.022474</td>\n",
       "      <td>{'clf__max_depth': 4, 'clf__max_features': 0.5, 'imp__strategy': 'median'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.867409</td>\n",
       "      <td>0.028762</td>\n",
       "      <td>{'clf__max_depth': 4, 'clf__max_features': 1, 'imp__strategy': 'mean'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.862605</td>\n",
       "      <td>0.029202</td>\n",
       "      <td>{'clf__max_depth': 4, 'clf__max_features': 1, 'imp__strategy': 'median'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.864299</td>\n",
       "      <td>0.029064</td>\n",
       "      <td>{'clf__max_depth': 5, 'clf__max_features': 0.5, 'imp__strategy': 'mean'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.862607</td>\n",
       "      <td>0.029290</td>\n",
       "      <td>{'clf__max_depth': 5, 'clf__max_features': 0.5, 'imp__strategy': 'median'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.854032</td>\n",
       "      <td>0.022482</td>\n",
       "      <td>{'clf__max_depth': 5, 'clf__max_features': 1, 'imp__strategy': 'mean'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.855886</td>\n",
       "      <td>0.032119</td>\n",
       "      <td>{'clf__max_depth': 5, 'clf__max_features': 1, 'imp__strategy': 'median'}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_test_score  std_test_score  \\\n",
       "0   0.870452         0.016936         \n",
       "1   0.869132         0.032918         \n",
       "2   0.855942         0.028338         \n",
       "3   0.856940         0.025588         \n",
       "4   0.864322         0.031108         \n",
       "5   0.866819         0.022474         \n",
       "6   0.867409         0.028762         \n",
       "7   0.862605         0.029202         \n",
       "8   0.864299         0.029064         \n",
       "9   0.862607         0.029290         \n",
       "10  0.854032         0.022482         \n",
       "11  0.855886         0.032119         \n",
       "\n",
       "                                                                        params  \n",
       "0   {'clf__max_depth': 3, 'clf__max_features': 0.5, 'imp__strategy': 'mean'}    \n",
       "1   {'clf__max_depth': 3, 'clf__max_features': 0.5, 'imp__strategy': 'median'}  \n",
       "2   {'clf__max_depth': 3, 'clf__max_features': 1, 'imp__strategy': 'mean'}      \n",
       "3   {'clf__max_depth': 3, 'clf__max_features': 1, 'imp__strategy': 'median'}    \n",
       "4   {'clf__max_depth': 4, 'clf__max_features': 0.5, 'imp__strategy': 'mean'}    \n",
       "5   {'clf__max_depth': 4, 'clf__max_features': 0.5, 'imp__strategy': 'median'}  \n",
       "6   {'clf__max_depth': 4, 'clf__max_features': 1, 'imp__strategy': 'mean'}      \n",
       "7   {'clf__max_depth': 4, 'clf__max_features': 1, 'imp__strategy': 'median'}    \n",
       "8   {'clf__max_depth': 5, 'clf__max_features': 0.5, 'imp__strategy': 'mean'}    \n",
       "9   {'clf__max_depth': 5, 'clf__max_features': 0.5, 'imp__strategy': 'median'}  \n",
       "10  {'clf__max_depth': 5, 'clf__max_features': 1, 'imp__strategy': 'mean'}      \n",
       "11  {'clf__max_depth': 5, 'clf__max_features': 1, 'imp__strategy': 'median'}    "
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make sure the entire params column will display\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "#Zip gs.cv_results_ into a table \n",
    "cv_results = zip(gs.cv_results_['mean_test_score'], gs.cv_results_['std_test_score'], gs.cv_results_['params'])\n",
    "#Convert table into a dataframe\n",
    "df = pd.DataFrame(cv_results, columns = ['mean_test_score', 'std_test_score','params'])\n",
    "#Sort by mean test score and display\n",
    "df.sort_values(by=['mean_test_score'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8704520863911731"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4FOX2wPHvSSCELk1FepPeQ5MO0lEQkCq92AtgQ68F9WIXfwpKVbwqIqKIKAqXLkiX0KVfCAgKSIf08/tjNmEJyWYJ2WzK+TzPPuyUnTk7WfbsvO/MeUVVMcYYY5IS4O8AjDHGpG+WKIwxxnhkicIYY4xHliiMMcZ4ZInCGGOMR5YojDHGeGSJwnhNRPqJyCJ/x5GeiMgFESnrh/2WFhEVkWxpvW9fEJEdItIiBa+zz2QasESRQYnI/0TksuuL6riIzBCRPL7cp6p+qaptfbkPdyJyh4gsFZHzInJWROaLSJW02n8i8SwXkWHu81Q1j6oe8NH+bheRb0TkpOv9bxWRUSIS6Iv9pZQrYZW/kW2oalVVXZ7Mfq5Jjmn9mcyqLFFkbHepah6gFlAbGOPneFIksV/FItIIWATMA24DygBbgNW++AWf3n6Zi0g5YB0QBlRX1fzAvUAIkDeV9+W3957ejrtJgqraIwM+gP8Bd7pNvwX85DadA3gHOAz8BUwCcrot7wKEAueA/UB71/z8wHTgGHAUeA0IdC0bBKxyPZ8EvJMgpnnAKNfz24BvgRPAQeAxt/VeBuYAX7j2PyyR9/cr8FEi838G/uN63gI4AjwHnHQdk37eHAO31z4DHAc+BwoAP7piPu16Xty1/r+BGCAcuABMcM1XoLzr+QxgIvATcB7ni76cWzxtgd3AWeAjYEVi79217hfuf89Elpd27Xug6/2dBJ53W14fWAOccf0tJwBBbssVeBjYCxx0zfs/nMR0DtgENHVbP9B1nPe73tsmoASw0rWti67j0su1fmecz9cZ4DegRoLP7jPAViACyIbb59kV+0ZXHH8B77nmH3bt64Lr0Qi3z6RrnarAf4F/XK99zt//VzPDw+8B2COFf7ir/2MVB7YB/+e2/H3gB6Agzi/Q+cDrrmX1XV9WbXDOKosBlVzLvgcmA7mBm4H1wP2uZfH/KYFmri8VcU0XAC7jJIgA1xfJi0AQUBY4ALRzrfsyEAV0da2bM8F7y4Xzpdwykfc9GDjmet4CiAbew0kKzV1fWBW9OAZxr33T9dqcQCGgu2v/eYFvgO/d9r2cBF/sXJso/nEd32zAl8As17LCri++bq5lj7uOQVKJ4jgw2MPfv7Rr31NdsdfE+dKt7FpeF2jo2ldpYBfwRIK4/+s6NnHJ8z7XMcgGjHbFEOxa9hTOZ6wiIK79FUp4DFzTdYC/gQY4CWYgzuc1h9tnNxQn0eR0mxf3eV4D9Hc9zwM0TPCes7ntaxBXPpN5cZLiaCDYNd3A3/9XM8PD7wHYI4V/OOc/1gWcX3cKLAFuci0TnC9M91+zjbjyy3EyMD6Rbd7i+rJxP/PoAyxzPXf/Tyk4v/CauaaHA0tdzxsAhxNsewzwqev5y8BKD++tuOs9VUpkWXsgyvW8Bc6XfW635bOBF7w4Bi2AyLgvwiTiqAWcdpteTvKJYprbso7AH67nA4A1bssEJ9EmlSiicJ3lJbE87kuzuNu89UDvJNZ/ApibIO5WyXzGTgM1Xc93A12SWC9hovgYeDXBOruB5m6f3SGJfJ7jEsVKYCxQOIn3nFSi6ANs9uX/u6z6sPbBjK2rqi4WkebATJxfrWeAIji/ijeJSNy6gvPrDpxfcgsS2V4pIDtwzO11AThfaFdRVRWRWTj/OVcCfXGaS+K2c5uInHF7SSBOc1Kca7bp5jQQCxQF/kiwrChOM0v8uqp60W36EM5ZTXLHAOCEqobHLxTJBYzHSUYFXLPzikigqsZ4iNfdcbfnl3B+EeOKKf49u47fEQ/bOYXzXlO0PxG5HedMKwTnOGTDOctzd9XfQERGA8NcsSqQD+czBc5nZr8X8YDz9x8oIo+6zQtybTfRfScwFHgF+ENEDgJjVfVHL/Z7PTGa62Cd2ZmAqq7A+TX7jmvWSZxmoKqqepPrkV+djm9w/pOWS2RTYThnFIXdXpdPVasmseuvgB4iUgrnLOJbt+0cdNvGTaqaV1U7uoft4f1cxGl+uDeRxT1xzp7iFBCR3G7TJYE/vTgGicUwGqdppYGq5sNpXgMnwXiM2QvHcM6UnA062at40quzGKcZLKU+xkmyFVzv5TmuvI848e9HRJri9Bv0BAqo6k04zZNxr0nqM5OYMODfCf7+uVT1q8T2nZCq7lXVPjhNn28Cc1x/4+SO//XEaK6DJYrM432gjYjUUtVYnLbr8SJyM4CIFBORdq51pwODRaS1iAS4llVS1WM4Vxq9KyL5XMvKuc5YrqGqm3E6fqcBC1U17gxiPXBORJ4RkZwiEigi1USk3nW8n2dxfpU+JiJ5RaSAiLyG03w0NsG6Y0UkyPVl1xn4xotjkJi8OMnljIgUBF5KsPwvnP6WlPgJqC4iXV1X+jwM3Oph/ZeAO0TkbRG51RV/eRH5QkRu8mJ/eXH6RC6ISCXgQS/Wj8b5e2YTkRdxzijiTANeFZEK4qghIoVcyxIel6nAAyLSwLVubhHpJCJeXa0lIveJSBHX3zDuMxXjii2WpP8GPwK3isgTIpLD9blp4M0+jWeWKDIJVT0B/AenfR6cX4f7gLUicg7nF2pF17rrcTqFx+P8alyB01wATlt6ELATpwloDp6bQL4C7sRp+oqLJQa4C6eN/yDOr/tpOFdUeft+VgHtcDp/j+E0KdUGmqjqXrdVj7vi/BOn8/gBVY1rrkryGCThfZyO4ZPAWuCXBMv/D+cM6rSIfODte3G9n5M4Z0hv4TQrVcG5siciifX34yTF0sAOETmLc8a2EadfKjlP4jQHnsf54v46mfUX4lxRtgfnWIdzdfPQezj9P4twEtB0nGMFTp/TZyJyRkR6qupGnD6rCTh/m304fQneao/zni/gHPPeqhquqpdwrj5b7dpXQ/cXqep5nAs07sL5XOwFWl7Hfk0S4q5YMSbDcd3J+4WqemrCSZdEJADn8tx+qrrM3/EY44mdURiTRkSknYjcJCI5uNJnsNbPYRmTLJ8lChH5RET+FpHtSSwXEflARPa5ShPU8VUsxqQTjXCuyjmJ0zzSVVUv+zckY5Lns6YnEWmGc53/f1S1WiLLOwKP4lxr3gDnZjHreDLGmHTGZ2cUqroS5y7VpHTBSSKqqmuBm0TEm+vGjTHGpCF/3nBXjKuvqjjimncs4YoiMgIYAZA7d+66lSpVSpMAjTEmI/jrXDh/n0/0Ajqiz/5NbMRFiI05qapFUrJ9fyaKhDf/QBI31KjqFGAKQEhIiG7cuNGXcRljTIYx7dcDvPbTLp4IKcETbSoAxJU3QUT4bPoUTp44wbtv/PtQSvfhz0RxBOeW+zjFca6FN8aYdO2fi5E89OUmLkRE+zWO2FjYeewcHavfyrhu1QkMEI4ePcqDDz5Ir1696NevH8+OehyAd9/4d4r3489E8QPwiKteUAPgrOvOYGOMSdcOnLjA2gP/UKvETRTKHeTXWBqULcizHSoRIDB16lSefPJJoqKi6NSpU6rtw2eJQkS+wqnQWdhV/OwlnIJzqOoknKJ0HXHu2ryEc6ewMSYLiI1Vdh0/R1RMxrzhd+/fFwAY3fZ2mlZIUbN/qtq/fz/Dhw9n2bJltGzZkqlTp1KuXOqVvfJZonAV9fK0PG7gFGNMFhIbq4yaHcr3oRm/pTlXUPoYlXbbtm1s2rSJKVOmMGzYMNwqJqcKKzNujEkzqsrY+Tv4PvRPHmhejgZlCvo7pBTLFRRI7RIFkl/RR7Zv387vv//OgAED6Nq1KwcOHKBQoULJvzAFLFEYY9LMpBUH+GzNIUY0K8sz7Sum+i/frCAyMpJx48Yxbtw4brnlFnr27ElwcLDPkgRYrSdjTBr6Ycuf1C1VgDEdKlmSSIF169ZRp04dxo4dS69evdi8eTPBwcE+36+dURhjUkV4VPKDAKoqBXMHWZJIgaNHj9K0aVNuueUWfvzxx1S9qik5liiMMSlyLjyKNftPsWrvSVbtO8nBkxeTfxFQpnDu5Fcy8fbs2cPtt99OsWLF+Prrr2ndujX58uVL/oWpyBKFMcYrUTGxbD58hlV7T/DrvpNsCTtDrDqdug3KFOSe2sXIHph8a3bLSv6/nDQjOHPmDE8//TTTpk1j+fLlNGvWjHvuuccvsViiMMZc48T5COZv+ZNYVSKiY/n90GnWHjjFxcgYAgRqFL+Jh1uWp0n5wtQuWYCgbNbdmZp++OEHHnzwQY4fP85TTz1FvXrXM4pw6rNEYYy5xuyNYby9cHf8dOlCubinTjGalC9Mo7KFyZ8rux+jy9yGDRvG9OnTqV69OvPmzSMkJMTfIVmiMMZcKybWuWN6y4ttyRYo5M5hXxW+5F7ELyQkhFKlSvHMM88QFOTf8iBx7K9vTCaz+/h5Hv3qdyKiY1O8jTOXogDIE5yNwAC7QsmXwsLCeOCBB+jduzf9+/fngQce8HdI17BEYUwm88fxc+z56wJ3Vr6ZPDdwJlC2SB5LEj4UGxvL5MmTeeaZZ4iJifFbR7U3LFGYNBURHcOmQ6fjmzZM6vvj+HkAnutYmbJF8vg5GpOYvXv3MmzYMFauXMmdd97JlClTKFOmjL/DSpIlCpOmZq0P46Ufdvg7jCzB+hXSr507d7J161Y++eQTBg0alO5vQLRPkklTFyOdgV5mDmtgl1T60E25grgln+9LOxjvbdmyhdDQUAYOHEiXLl04cOAABQr4r6jg9bBEYQB4f/Eedh075/P9HDjh3L1bp1QBgrOnjxLNxvhSREQEr732Gm+88QZFixalV69eBAcHZ5gkAZYojMvHy/eTO0c2bs6bw6f7CQwQ2lW9hRx2NmGygDVr1jB06FB27drFgAEDeO+999KkiF9qs0Rh4t0bUpwxHSr7OwxjMoWjR4/SvHlzbr31VhYsWECHDh38HVKK2c86Y4xJRbt27QKgWLFizJ49mx07dmToJAGWKIwxJlWcPn2aIUOGUKVKFX799VcAunbtSt68ef0c2Y2zpqcsaPW+k6w7+M9V86LtvgZjUmzu3Lk89NBDnDhxgjFjxvi9iF9qs0SRxSzccZwHv9hEwrwQIFDObs4y5roNGTKETz/9lFq1avHTTz9Rp04df4eU6rxKFCKSDygKXAbCNK6ClclQftt3kkdnbqZG8Zv4clgDuyHLmBRyL+LXsGFDKlSowJNPPkn27Jmzqm6S3xQikhd4EOgL5AFOAsFAIRFZBXykqr+mSZTmhv1zMZLh/9lImcK5mTG4niUJY1Lo0KFD3H///fTt25cBAwYwYsQIf4fkc546s+cCJ4DWqlpeVRuqai2gJPA+0EtEhqRFkObG/XnmMhcjYxjZ5nZuypU+Shcbk5HExsYyceJEqlWrxqpVq4iKivJ3SGkmyZ+VqnpnEvMVWOd6mHRqS9gZjp65HD8d9s8lwOmLMMZcn927dzNs2DBWrVpF27ZtmTx5MqVLl/Z3WGkm2fYHEZkFfAossr6JjCE2Vrl38hoiExmPwM4mjLl+u3fvZseOHcyYMYMBAwak+yJ+qc2bhuoZwBBggoh8DcxQ1X0+jcrcsMjoWPo3LMV9DUvFz8uZPZCShXL5MSpjMo7NmzcTGhrK4MGDufvuuzlw4AA33XSTv8Pyi2QThar+AvwiIgWAfsAyETkITAW+UtVoH8dokjHt1wMs/ePv+Om4877CeXJQ8daMf7OPMWkpPDycV155hbfeeotixYrRp08fgoODs2ySAC/vzHYlib5Af2ArMBm4A/jFd6EZb32z8Qg7j50jKiaWqJhYomNjaVCmIE0qFPJ3aMZkKKtXr6ZWrVq8/vrrDBgwgNDQ0AxZxC+1edNHMRuoDswEuqvqEdeiL0Vksy+DM95rWKYQk/rX9XcYxmRYR48epWXLlhQrVoyFCxfStm1bf4eUbnjTRzEN+K97R7aIZFPVaFWt7bvQjDHG93bu3EmVKlUoVqwY3377LS1btiRPHqtS4M6bpqc3Ernaab0vgjHGmLTyzz//MGjQIKpWrcrKlSsBuOuuuyxJJMLTndk345TtyCki1YG468HyAXbpTBr7ZNVBwk5fSnTZX+fDKVM4dxpHZEzG9e233/Lwww9z6tQpnn/+eerXr+/vkNI1T01PnXAuiy0OfOQ2/zzwgi+DMle7GBHNKz/uJChbQJIjw1Uvnj+NozImYxo0aBCfffYZderU4ZdffqFWrVr+Dind83Rn9qfApyLSU1Vnp2FMJoG4dr+n2lZkeLOyfo3FmIzIvYjfHXfcQeXKlRk9ejTZslnNM294anrqo6pfAUVF5LGEy1X1g+Q2LiLtgf8DAoFpqvpGguUlgc+Am1zrPKuqC67vLWRs7/13D5OW7/e4jhL3IU+LiIzJXA4ePMiIESO47777GDhwYJYo4pfaPKXTAq5/C6dkwyISCEwE2gBHgA0i8oOq7nRb7V/AbFX9WESqAAuA0inZX0b1x7Fz5A3ORs96JTyuly1AuKvmbWkUlTEZX0xMDBMnTmTMmDEEBATQr18/f4eUYXlqeorrlxivqv8ktZ4H9YF9qnoA4mtGdQHcE4XidI4D5Af+TMF+MowzlyJZvOtvYt1GDTpy+jJF8ubgmfaV/BiZMZnLrl27GDp0KGvWrKFDhw5MmjSJkiVL+jusDMubBroNIrIb+BqYq6rnvNx2MSDMbfoI0CDBOi8Di0TkUSA3kGjFWhEZAYwAMvQf+8t1h3l74e5r5jetkKKTNmNMEvbt28fu3bv5/PPP6devX5Yr4pfavKn1VE5E7gB6A2NFJBSYpaqzknlpYn+ZhPdj9MEpMviuiDQCPheRaqp6VdlTVZ0CTAEICQnJsBVsI1zVXFc/2+qq+YXzWEVXY27Upk2b2LJlC0OGDOGuu+7i4MGD5MuXL/kXmmR51eWvqr8Bv4nIyziDFn0JJJcojgDuDe/FubZpaSjQ3rWPNSISjNMn8jeZyNj5O9j4v9McPxcOQLGbcvo5ImMyj8uXLzN27FjeeecdSpQoQd++fQkODrYkkYqSvTNbRPKISD8RmY9zR/YJnIKAydkAVBCRMiIShHNG8kOCdQ4DrV37qYwz1OqJ64g/Q5i/5U/+uRhJ9WL5ud8ubzUm1axcuZKaNWvy5ptvMmjQIDZv3mxF/HzAmzOK7cB84K3rGSNbVaNF5BFgIc6lr5+o6g4ReQXYqKo/AKOBqSIyEqdZalBGHxzpQkQ0+/++cNW8qBilXdUi/Pue6n6KypjM5+jRo7Ru3ZoSJUqwePFiWrdu7e+QMi1vEkXZhH0G3nLdE7EgwbwX3Z7vBBqnZNvp1dNztrBg2/Fr5ucKCvRDNMZkPtu2baN69eoUK1aMuXPn0rJlS3LnthI2vuTphrt3VXU08K2IXPMrX1W7+TSyDOp8eDTliuTm+U6V4+cJQt3SBTy8yhiTnJMnTzJy5Ei++OILVqxYQbNmzejcubO/w8oSPJ1RfO36d0JaBJKZ5M+ZnVaVbvF3GMZkCqrKN998wyOPPMLp06d56aWXaNAg4ZX2xpc83XAXV0q8sqpelSxcfQ9LfBmYMcYADBw4kM8//5yQkBCWLFlC9erW15fWvOmjGMK1ZxVDE5mXJUVEx+De/R4Tm6H74o1JF9yL+DVv3pwaNWrwxBNPWBE/P/HUR9EL55LWMiLynduivMAZXweWEXy+9hAvfL/9mvn1Sxf0QzTGZA4HDhxg+PDh3HfffQwePJihQ4f6O6Qsz1N6Xg+cwrlRbqLb/POAjZUNHD51keyBwsg2t181v1HZQn6KyJiMKyYmhg8//JDnn3+ewMBABgwY4O+QjIunPoqDwEFgcdqFk/FkDwzgoRbl/R2GMRnazp07GTJkCOvWraNTp05MmjSJ4sWL+zss4+Kp6WmFqjYXkdNcXaNJAFVVa18xxqSKgwcPsn//fmbOnEnv3r2tiF8646npqaXrXyttaoxJdRs2bCA0NJThw4fTqVMnDhw4QN68ef0dlkmEp6anuLuxSwB/qmqkiDQBagBfAN6WG89wzoVH0X/6es5djvK43qkLEWkUkTGZx6VLl3jxxRcZP348pUqVon///gQHB1uSSMe8udbse6CeiJQD/gP8BMwEMu0tkUdPX2ZL2BnqlS5A0fyeK71Wvc0qVBrjreXLlzNs2DD279/P/fffz5tvvmlF/DIAbxJFrKpGiUg34H1V/UBEssRVT0OblKF9taL+DsOYTOHIkSO0adOGUqVKsXTpUlq2bJn8i0y6kGyZcSBaRO4F+gM/uuZl911IxpjMZMuWLQAUL16cefPmsXXrVksSGYw3iWIITsf2W6p6QETKAF/5NixjTEZ34sQJ+vbtS61atVixYgUAHTt2JFeuXH6OzFwvb4ZC3Q485DZ9EPi3L4MyxmRcqsqsWbN47LHHOHv2LGPHjqVRo0b+DsvcgGQThYg0BF4ESrvWj7uP4nZPrzPGZE39+/fnyy+/pEGDBkyfPp2qVav6OyRzg7zpzP4UeBrYBMT4Nhz/UlX+uRjJ6UuR/g7FmAwlNjYWEUFEaNmyJXXr1uWxxx4jMNAG7MoMvEkU51R1vs8jSQcmLtvHO4v2xE9nD/SmC8eYrG3fvn0MHz6c/v37M2TIECvilwl58024VEReF5F6IlIj7uHzyPzg2NlwcgcF8mqXqrzdowaNy9tN6cYkJTo6mnfeeYfq1auzefNmgoKC/B2S8RFvziiaJPgXnNpPzVI/nLTzz8VIvlp/mMjoK8OBbz1ylpxBgfRvVNp/gRmTAWzfvp3BgwezceNGunTpwkcffcRtt93m77CMj3hz1VPTtAgkrS3ccZy3F+6+Zn7Dslbr0JjkHD58mEOHDjFr1ix69uxpRfwyOW+ueioCvAYUU9XOIlIFqK+qM3wdnC/FjUS3/vnW3JzXSggYk5x169axZcsWRowYQceOHTlw4AB58uTxd1gmDXjTRzEDWIFTHBBgLzDaVwEZY9KXixcvMmrUKBo1asRbb71FRIRTDNOSRNbhTaK4WVVnArEAqhpFJr9M1hjjWLp0KTVq1GD8+PE88MAD/P777+TIkcPfYZk05k1n9kURKYhr8CIRqYczHKoxJhM7cuQI7dq1o0yZMqxYsYJmzTL09SvmBniTKJ4E5gNlRWQFUAzo4dOojDF+s3nzZmrXrk3x4sWZP38+zZs3J2dOz+X2TeaWbNOTqm7EKQrYHHgcqKKqob4OzBiTtv766y969epFnTp14ov4tW/f3pKE8Thmdl3giKr+5RrdrhrQDTgkIq+o6pk0izIVfbLqIEv++ItjZ8P9HYox6YKq8uWXX/L4449z4cIFXnvtNe644w5/h2XSEU9nFFOAaADXEKjvALOBcNeyDGn2xjC2Hz1HwVxBdKpelIK57G5Sk7X17duX/v37U7FiRUJDQ3n++efJnt2GnDFXeOqjyKaqp1zPewNTVPVr4GsR2eL70HynQZmCTBkQ4u8wjPEb9yJ+bdu2pVGjRjz88MNWxM8kytMZRaCIxH1qWgNL3Zf5LiRjjC/t2bOHli1b8sknnwAwePBgq/RqPPKUKGYDy0TkWyAS+BVARMoB59IgNmNMKoqOjuatt96iZs2abN261TqpjdeSbHpS1VdEZClQFPhFVeOq52UHHkuL4IwxqWPr1q0MGTKETZs2cc899zBx4kSKFi3q77BMBuHpqqdcqroq4XxV/SPBOpd8FZwxJnUcOXKEsLAwvvnmG7p3725F/Mx18dT09KOIvCkid4hIfNU8ESkpIgNFZAHQ2dPGRaS9iOwWkX0i8mwS6/QUkZ0iskNEZqbsbRhjEvrtt9+YNGkSQHwRvx49eliSMNfNU6JoDazGuclun4icEZF/gDk442cPV9XZSb3Y1RE+EegAVAH6uCrPuq9TARgDNFbVqsATN/BejDHAhQsXePzxx2nSpAnvvvtufBG/3Llz+zkyk1F56qNQ4AfXIyXqA/tU9QCAiMwCugA73dYZDkxU1dOuff6dwn0ZY4BFixYxYsQIDh8+zMMPP8y4ceOsiJ+5Yd7UekqpYkCY2/QRoEGCdW4HEJHVOJfcvqyqvyTckIiMAEYAlCxZ0ifBGpPRhYWF0alTJ8qVK8fKlStp0qRJ8i8yxgvelBlPqcQaQjXBdDagAtAC6ANME5GbrnmR6hRVDVHVkCJFiqR6oMZkZJs2bQKgRIkSLFiwgNDQUEsSJlX5MlEc4cpgRwDFgT8TWWeeqkap6kFgN07iMMYk4/jx49x7772EhITEF/Fr06YNwcE2YqNJXR4ThYgE3kC5jg1ABREpIyJBOGVAEvZ3fI9TmRYRKYzTFHUghftL0plLkfSbtpYuE1Zx8OTF1N68MWlKVfnss8+oUqUK8+fPZ9y4cVbEz/iUx0ShqjHAThEpdr0bVtVo4BFgIbALmK2qO0TkFRG527XaQuCUiOwElgFPudWXSjX7T1xk9b5TKNCoXCHuqX3db8eYdKN3794MGjSIKlWqEBoaypgxY6yIn/EpbzqzCwO7RGQNEP9zXFW7JfdCVV0ALEgw70W35wqMcj18bnTbijS/3fo4TMbjXsSvY8eONG3alIceeoiAAF+2Hhvj8CZRvOHzKIwxSfrjjz8YNmwYgwYNYtiwYQwcONDfIZksxpsR7pYAW3BqPGUHtrjmGWN8KCoqinHjxlGzZk127txJnjx5/B2SyaKSPaMQke7AeJzqsQJMEpGRqjrX18HdqA3/+4dvNoZx8kKkv0Mx5rqEhoYyePBgQkND6dGjBx9++CG33nqrv8MyWZQ3TU8vAvVU9S8AEbkFWASk+0Tx1brDfB96lFvyBVO2SG7KFrYSBiZjOH78OMePH+fbb7+lW7dkuwON8SlvEkVAXJJwOYFv779IVcUK5OTXp1v5OwxjkrVq1Sq2bt3KQw89RPv27dm/fz+5cuXyd1jGePWFv0hEFojIfSJyH869EAt9HJcxWcb58+d55JFHaNq0Ke+//358ET9LEia98CZRPAnMwCny1wD4DHjKhzEZk2UsXLjF5xkqAAAgAElEQVSQatWq8dFHH/H444/z+++/WxE/k+4k2/TkutdhtuthjEklYWFhdO7cmfLly7Nq1Sq7u9qkWxmmr8GYzEBVWb9+PeAU8fv555/ZvHmzJQmTrlmiMCaNHDt2jO7du9OgQYP4In533nmnFfEz6Z5X41G4ivqVVNV9Po4nxdYf/Ifn524jJvZKJfO/z0dQILfVwDH+parMmDGDUaNGER4ezptvvknjxo39HZYxXvPmhrtOwHtAEFBGRGoBL6nqPb4O7npsCTvD3r8v0KHarWQLdE6UqgINyhT0b2Amy+vZsydz5syhadOmTJs2jdtvv93fIRlzXbw5o3gF52qnZQCqGioi5X0a1Q14+96a5Mnhy4H7jEleTEwMIkJAQAB33XUXrVq14v7777cifiZD8uZTG6WqZxLMSzhSnTHGZdeuXTRt2pTp06cDMGDAAB588EFLEibD8uaTu0tEegIBrkGI3gfW+jguYzKcqKgoXnvtNWrVqsXu3bvJnz+/v0MyJlV4kygeAeoCscB3QDjwuC+DMiaj2bx5MyEhIbzwwgvcc8897Nq1i549e/o7LGNShTeN+e1U9RngmbgZItINJ2kYY4C//vqLkydP8v3339OlSxd/h2NMqvLmjOJficx7PrUDMSajWblyJRMnTgSgffv27Nu3z5KEyZSSPKMQkXZAe6CYiLzntigfTjOUMVnSuXPnePbZZ/n444+5/fbbGTZsGDly5CBnzpz+Ds0Yn/B0RvE3sB2nT2KH22MR0MH3oRmT/ixYsICqVasyefJkRo0aZUX8TJaQ5BmFqm4GNovIl6oanoYxGZMuhYWF0aVLFypWrMicOXNo0KCBv0MyJk1405ldTET+DVQB4ovSqKrdXmoyPVVl3bp1NGzYkBIlSrBo0SIaN25MUFCQv0MzJs1405k9A/gUZ7zsDjjlxmf5MCZj0oU///yTrl270qhRo/gifi1btrQkYbIcb84ocqnqQhF5R1X3A/8SkV99HZgnYf9cov/0dVyMjImfdykiGnCymTE3QlWZPn06Tz75JBEREbzzzjtWxM9kad4kiggREWC/iDwAHAVu9m1Ynh06dYn/nbrEnZVvpkjeKyWaSxTMSW6r82RuUI8ePfjuu+9o3rw506ZNo3z5dFvazJg04c236kggD/AY8G8gPzDEl0F56/7m5ahX2qrDmhvnXsSva9eutG3bluHDh1t9JmPwbijUda6n54H+ACJS3JdBGZOWtm/fzrBhwxg6dCjDhw+nf//+/g7JmHTF488lEaknIl1FpLBruqqI/AcrCmgygcjISMaOHUudOnXYv38/BQoU8HdIxqRLSSYKEXkd+BLoB/wiIs/jjEmxBbBLY02GtmnTJurWrcvLL7/Mvffey86dO+nRo4e/wzImXfLU9NQFqKmql0WkIPCna3p32oRmjO+cOnWKM2fOMH/+fDp37uzvcIxJ1zwlinBVvQygqv+IyB+WJExGtmzZMrZt28Zjjz1G27Zt2bt3L8HBwcm/0JgszlMfRVkR+c71mAuUdpu2EuMmwzh79iz3338/rVq14uOPPyYiIgLAkoQxXvJ0RtE9wfQEXwZijC/Mnz+fBx54gOPHj/Pkk08yduxYK+JnzHXyVBRwSVoGYkxqCwsLo3v37lSqVInvv/+eevXq+TskYzIku5vIZCqqym+//QYQX8Rv48aNliSMuQE+TRQi0l5EdovIPhF51sN6PURERSTEl/GYzO3IkSPcfffdNG7cOL6IX4sWLayInzE3yOtEISLX1bArIoHARJyKs1WAPiJSJZH18uKUB1mXcJkx3oiNjWXy5MlUqVKFJUuW8N5779GkSRN/h2VMppFsohCR+iKyDdjrmq4pIh96se36wD5VPaCqkTilyRMbUPhV4C2ckfSMuW7du3fngQceoF69emzfvp2RI0cSGBjo77CMyTS8OaP4AOgMnAJQ1S1ASy9eVwwIc5s+4poXT0RqAyVU9UdPGxKRESKyUUQ2njhxwotdm8wuOjqa2Fhn6Pbu3bszdepUFi9eTNmyZf0cmTGZjzeJIkBVDyWYF5PomldLbGgIjV8oEgCMB0YntyFVnaKqIaoaUqRIES92bTKzrVu30qhRI6ZOnQrAfffdx7Bhw3Cq4RtjUps3iSJMROoDKiKBIvIEsMeL1x0BSrhNF8cpAxInL1ANWC4i/wMaAj9Yh7ZJSkREBC+99BJ169bl0KFD2I8GY9KGN+NRPIjT/FQS+AtY7JqXnA1ABREpgzPYUW+gb9xCVT0LFI6bFpHlwJOqutHb4E3WsWHDBgYNGsTOnTvp378/48ePp1ChQv4Oy5gswZtEEa2qva93w6oaLSKPAAuBQOATVd0hIq8AG1X1h+vdpsm6Tp8+zYULF1iwYAEdOnTwdzjGZCneJIoNIrIb+Br4TlXPe7txVV0ALEgw78Uk1m3h7XZN1rB06VK2bdvG448/Ttu2bdmzZ4+V3zDGD5Lto1DVcsBrQF1gm4h8LyLXfYZhjLfOnDnD8OHDad26NZMnT44v4mdJwhj/8OqGO1X9TVUfA+oA53AGNDIm1c2bN48qVarwySef8PTTT7Np0yZLEMb4WbJNTyKSB+dGud5AZWAecIeP4zJZ0OHDh7n33nupXLkyP/zwAyEhdgGcMemBN30U24H5wFuq+quP4zFZjKqyatUqmjZtSsmSJVm8eDENGza0+kzGpCPeND2VVdVHLUmY1Hb48GE6depEs2bN4ov4NWvWzJKEMelMkmcUIvKuqo4GvhURTbhcVbv5NDKTacXGxjJp0iSeeeYZVJUPPvjAivgZk455anr62vWvjWxnUlW3bt2YN28ebdq0YcqUKZQuXdrfIRljPPA0wt1619PKqnpVsnDdSOeXEfBOXohg4Y7j/ti1uQHR0dEEBAQQEBBAr1696NKlC4MGDbL6TMZkAN70UQxJZN7Q1A7EW8fOhvP52kNkDxQK57HLJjOCLVu20KBBA6ZMmQJAnz59GDx4sCUJYzIIT30UvXAuiS0jIt+5LcoLnPF1YEnJkS2A0BfbEJQtgFxB3ly0ZfwlPDyc1157jTfffJOCBQty6623+jskY0wKePqmXY8zBkVxnJHq4pwHNvsyKE8E4aZcdlVMerd+/XoGDhzIH3/8wcCBA3nvvfcoWLCgv8MyxqSApz6Kg8BBnGqxxlyXc+fOcfnyZX755RfatWvn73CMMTdAVK+58tVZILJCVZuLyGncBhzCGZBIVdUvPw/zl6ikZ8P+8MeuTTIWLVrEjh07GDlyJOCMH2HlN4xJH0Rkk6qmqNyBp87suOFOCwNF3B5x08YATgnwwYMH065dO6ZPn25F/IzJZJJMFKoa63paAghU1RigEXA/kDsNYjMZwHfffUeVKlX4/PPPGTNmDBs3brQEYUwm483lsd/jDINaDvgPTmHAmT6NymQIhw8fpnfv3hQtWpQNGzYwbtw4goOD/R2WMSaVeZMoYlU1CugGvK+qjwLFfBuWSa9UNb4uU8mSJVm6dCnr1q2jdu3afo7MGOMr3iSKaBG5F+gP/Oial913IZn06tChQ3To0IEWLVrEJ4smTZqQPbt9HIzJzLy9M7slTpnxAyJSBvjKt2GZ9CQ2NpYJEyZQtWpVVq1axYcffkjTpk39HZYxJo0keXnsVSuJZAPKuyb3qWq0T6PywC6PTXt333038+fPp127dkyePJlSpUr5OyRjzHW6kctjvRnhrinwOXAU5x6KW0Wkv6quTskOTcYQFRVFYGAgAQEB9OnThx49etC/f3+rz2RMFuRN09N4oKOqNlbVO4BOwP/5NizjT7///jv169dn0qRJgFPEb8CAAZYkjMmivEkUQaq6M25CVXcBVmwpE7p8+TJjxoyhfv36HD9+nBIlSvg7JGNMOuBN+dXfRWQyTvMTQD/8WBTQ+MbatWsZOHAge/bsYciQIbzzzjsUKFDA32EZY9IBbxLFA8BjwNM4fRQrgQ99GZRJexcvXiQqKor//ve/3Hnnnf4OxxiTjni86klEqgPlgB2qujfNovLArnpKPb/88gs7duxg9OjRAERGRhIUZK2KxmRGPikKKCLP4ZTv6Af8V0QSG+nOZECnTp1i4MCBdOjQgc8++4zIyEgASxLGmER56szuB9RQ1XuBesCDaROS8RVVZc6cOVSpUoWZM2fyr3/9iw0bNliCMMZ45KmPIkJVLwKo6gkR8eYKKZOOHT58mL59+1KjRg0WLVpEzZo1/R2SMSYD8JQoyrqNlS1AOfexs1W1m08jM6lCVVm2bBmtWrWiVKlSLF++nPr165Mtm403bozxjqdvi+4Jpif4MhCT+g4ePMiIESNYvHgxy5cvp3nz5txxxx3+DssYk8F4GjN7SVoGYlJPTEwMEyZM4LnnniMwMJCPP/7YivgZY1LM2h8yoS5duvDTTz/RsWNHJk2aZHdYG2NuiCWKTMK9iF///v3p06cPffv2tfpMxpgb5vWVTCJy3QMhi0h7EdktIvtE5NlElo8SkZ0islVEloiI1a9OgY0bNxISEsLHH38MQK9evejXr58lCWNMqkg2UYhIfRHZBux1TdcUkWRLeIhIIDAR6ABUAfqISJUEq20GQlS1BjAHeOs648/SLl++zDPPPEODBg04ceKEjRNhjPEJb84oPgA6A6cAVHULzoh3yamPM8jRAVWNBGYBXdxXUNVlqnrJNbkWKO5t4FndmjVrqFmzJm+99RZDhgxh586ddO7c2d9hGWMyIW/6KAJU9VCCZowYL15XDAhzmz4CNPCw/lDg58QWiMgIYARArqLlvNh15nf58mViY2NZvHgxrVu39nc4xphMzJtEESYi9QF1NSc9Cuzx4nWJNZAnWoFQRO4DQoDmiS1X1SnAFHCKAnqx70xpwYIF7Nixg6eeeopWrVqxa9cusmfP7u+wjDGZnDdNTw8Co4CSwF9AQ7yr+3QEcL8uszjwZ8KVRORO4HngblWN8GK7Wc7Jkye577776NSpE19++WV8ET9LEsaYtJBsolDVv1W1t6oWdj16q+pJL7a9AaggImVEJAjoDfzgvoKI1AYm4ySJv1PyBjIzVWXWrFlUrlyZ2bNn89JLL7F+/Xor4meMSVPJNj2JyFQSaTJS1RGeXqeq0SLyCLAQCAQ+UdUdIvIKsFFVfwDeBvIA37j6QA6r6t3X/zYyp8OHDzNw4EBq1qzJ9OnTqV69ur9DMsZkQR4HLgIQkV5uk8HAPUCYqj7qy8CSktkHLlJVlixZEj/K3Nq1a6lXrx6BgYF+jswYk5H5ZOCiOKr6tdvjM6Abzn0RJpXt37+f1q1b06ZNG1asWAFAw4YNLUkYY/wqJWNMlAHszq5UFBMTw3vvvUf16tXZtGkTkydPtiJ+xph0w5s+itNc6aMIAP4BrinHYVLurrvu4ueff6Zz5858/PHHFC9u9x0aY9IPj30U4vQwlwCOumbFanKdGj6WWfooIiMjyZYtGwEBAcyePZuYmBh69+5t9ZmMMT7hsz4KV1KYq6oxrkeWvdktNa1fv566devy0UcfAdCzZ0/69OljScIYky5500exXkTq+DySLODSpUuMHj2aRo0acfr0acqVs3Ikxpj0L8k+ChHJpqrRQBNguIjsBy7ilOZQVbXkcR1WrVrFwIEDOXDgAPfffz9vvvkm+fPn93dYxhiTLE+d2euBOkDXNIolU4sbWGjZsmW0aNHC3+EYY4zXPCUKAVDV/WkUS6Yzf/58du3axdNPP03Lli3ZuXMn2bLZoILGmIzF07dWEREZldRCVX3PB/FkCidOnODxxx/nq6++olatWjzxxBMEBQVZkjDGZEieOrMDceow5U3iYRJQVWbOnEnlypWZM2cOr7zyCuvWrbMifsaYDM3TT9xjqvpKmkWSCRw+fJjBgwdTu3Ztpk+fTtWqVf0dkjHG3DBPZxR2Ub8XYmNjWbhwIQClSpXi119/ZfXq1ZYkjDGZhqdEYeNrJmPv3r20atWK9u3bs3LlSgDq169vRfyMMZlKkolCVf9Jy0AykujoaN5++21q1KhBaGgo06dPtyJ+xphMyy7DSYHOnTuzcOFCunTpwkcffcRtt93m75CMH0VFRXHkyBHCw8P9HYoxBAcHU7x48VQdKjnZgYvSG38VBYyIiCB79uwEBAQwZ84cYmNjuffee60+k+HgwYPkzZuXQoUK2efB+JWqcurUKc6fP0+ZMmWuWubTgYuMM8pcnTp1mDhxIgA9evSgZ8+e9qVgAAgPD7ckYdIFEaFQoUKpfnZricKDixcvMnLkSO644w7Onz9PhQoV/B2SSacsSZj0whefReujSMKvv/7KwIEDOXjwIA899BCvv/46+fLl83dYxhiT5uyMIgnR0dFkz56dFStWMHHiREsSJl0LDAykVq1aVKtWjbvuuoszZ87EL9uxYwetWrXi9ttvp0KFCrz66qu4903+/PPPhISEULlyZSpVqsSTTz7pj7fg0ebNmxk2bJi/w/Do9ddfp3z58lSsWDH+3qqElixZQp06dahVqxZNmjRh3759gHOzbsuWLalduzY1atRgwYIFgHOhxMCBA6levTqVK1fm9ddfB5yBz5o1a0Z0dHTavDlVzVCPfMUrqq/MnTtXx40bFz8dFRXls32ZzGPnzp3+DkFz584d/3zAgAH62muvqarqpUuXtGzZsrpw4UJVVb148aK2b99eJ0yYoKqq27Zt07Jly+quXbtU1fnMT5w4MVVjS43/Rz169NDQ0NA03ef12LFjh9aoUUPDw8P1wIEDWrZsWY2Ojr5mvQoVKsR/XiZOnKgDBw5UVdXhw4frRx99FL+tUqVKqarql19+qb169VJV529XqlQpPXjwoKqqvvzyy/rFF18kGk9in0lgo6bwe9eanoC//vqLRx99lG+++YY6deowevRoK+JnUmTs/B3s/PNcqm6zym35eOku7+/0b9SoEVu3bgVg5syZNG7cmLZt2wKQK1cuJkyYQIsWLXj44Yd56623eP7556lUqRIA2bJl46GHHrpmmxcuXODRRx9l48aNiAgvvfQS3bt3J0+ePFy4cAGAOXPm8OOPPzJjxgwGDRpEwYIF2bx5M7Vq1WLu3LmEhoZy0003AVC+fHlWr15NQEAADzzwAIcPHwbg/fffp3Hjxlft+/z582zdupWaNWsCzgiRTzzxBJcvXyZnzpx8+umnVKxYkRkzZvDTTz8RHh7OxYsXWbp0KW+//TazZ88mIiKCe+65h7FjxwLQtWtXwsLCCA8P5/HHH2fEiBFeH9/EzJs3j969e5MjRw7KlClD+fLlWb9+PY0aNbpqPRHh3Dnn83H27Nn4S+s9zb948SLR0dFcvnyZoKCg+NaNrl27MmbMGPr163dDsXsjS38TqipffPEFTzzxBBcuXODf//43Tz31VKpef2xMWoqJiWHJkiUMHToUcJqd6tate9U65cqV48KFC5w7d47t27czevToZLf76quvkj9/frZt2wbA6dOnk33Nnj17WLx4MYGBgcTGxjJ37lwGDx7MunXrKF26NLfccgt9+/Zl5MiRNGnShMOHD9OuXTt27dp11XY2btxItWrV4qcrVarEypUryZYtG4sXL+a5557j22+/BWDNmjVs3bqVggULsmjRIvbu3cv69etRVe6++25WrlxJs2bN+OSTTyhYsCCXL1+mXr16dO/enUKFCl2135EjR7Js2bJr3lfv3r159tlnr5p39OhRGjZsGD9dvHhxjh49es1rp02bRseOHcmZMyf58uVj7dq1ALz88su0bduWDz/8kIsXL7J48WLAucJy3rx5FC1alEuXLjF+/HgKFiwIQLVq1diwYUOyf4fUkKUTxeHDhxk2bBghISFMnz49/leVMSl1Pb/8U9Ply5epVasW//vf/6hbty5t2rQBnB9DSV0Fcz1XxyxevJhZs2bFTxcoUCDZ19x7773x5Wx69erFK6+8wuDBg5k1axa9evWK3+7OnTvjX3Pu3DnOnz9P3rxXClQfO3aMIkWKxE+fPXuWgQMHsnfvXkSEqKio+GVt2rSJ/yJdtGgRixYtonbt2oBzVrR3716aNWvGBx98wNy5cwEICwtj79691ySK8ePHe3dw4Ko+nziJHd/x48ezYMECGjRowNtvv82oUaOYNm0aX331FYMGDWL06NGsWbOG/v37s337dtavX09gYCB//vknp0+fpmnTptx5552ULVuWwMBAgoKCrjlevpDlOrNjY2P5+eefAaeI3+rVq1m5cqUlCZOh5cyZk9DQUA4dOkRkZGT8PT9Vq1Zl48aNV6174MAB8uTJQ968ealatSqbNm1KdvtJJRz3eQmv3c+dO3f880aNGrFv3z5OnDjB999/T7du3QDn/+OaNWsIDQ0lNDSUo0ePXvOllzNnzqu2/cILL9CyZUu2b9/O/Pnzr1rmvk9VZcyYMfHb3rdvH0OHDmX58uUsXryYNWvWsGXLFmrXrp3ofQcjR46kVq1a1zzeeOONa9YtXrw4YWFh8dNHjhy5pmLDiRMn2LJlCw0aNACc5Pnbb78BMH36dHr27Bl/rMLDwzl58iQzZ86kffv2ZM+enZtvvpnGjRtf9feMiIggODj4mnhSW5ZKFHv27KFFixZ07NiRFStWABASEmJF/EymkT9/fj744APeeecdoqKi6NevH6tWrYpvyrh8+TKPPfYYTz/9NABPPfUU48aNY8+ePYDzxf3ee9eOSda2bVsmTJgQPx3X9HTLLbewa9eu+KalpIgI99xzD6NGjaJy5crxv94Tbjc0NPSa11auXDn+6iBwziiKFSsGwIwZM5LcZ7t27fjkk0/i+1COHj3K33//zdmzZylQoAC5cuXijz/+iG/+SWj8+PHxScb9kbDZCeDuu+9m1qxZREREcPDgQfbu3Uv9+vWvWqdAgQKcPXs2/lj/97//pXLlygCULFmSJUuWALBr1y7Cw8MpUqQIJUuWZOnSpagqFy9eZO3atfE/ak+dOkWRIkXSpKk8SySK6Oho3nzzTWrUqMG2bdv49NNPadasmb/DMsYnateuTc2aNZk1axY5c+Zk3rx5vPbaa1SsWJHq1atTr149HnnkEQBq1KjB+++/T58+fahcuTLVqlXj2LFj12zzX//6F6dPn6ZatWrUrFkzvu3+jTfeoHPnzrRq1YqiRYt6jKtXr1588cUX8c1OAB988AEbN26kRo0aVKlShUmTJl3zukqVKnH27FnOnz8PwNNPP82YMWNo3LgxMTExSe6vbdu29O3bl0aNGlG9enV69OjB+fPnad++PdHR0dSoUYMXXnjhqr6FlKpatSo9e/akSpUqtG/fnokTJ8b/AO3YsSN//vkn2bJlY+rUqXTv3p2aNWvy+eef8/bbbwPw7rvvMnXqVGrWrEmfPn2YMWMGIsLDDz/MhQsXqFatGvXq1WPw4MHUqFEDgGXLltGxY8cbjt0bWaLWU7t27Vi0aBHdunVj4sSJ3HrrrT6KzmRFu3btiv9laHxj/Pjx5M2bN93fS5GWunXrxuuvv07FihWvWZbYZ9JqPSUiPDw8/tfGiBEjmDNnDt9++60lCWMyoAcffJAcOXL4O4x0IzIykq5duyaaJHwhUyaK1atXU6tWrfgOve7du9O9e3c/R2WMSang4GD69+/v7zDSjaCgIAYMGJBm+8tUieLChQs89thjNG3alPDwcGsOMGkmozXhmszLF5/FTJMoVqxYQbVq1ZgwYQKPPPII27dvj7+W3BhfCg4O5tSpU5YsjN+pazyK1L5kNlPdcJcrVy5+/fXXa0oAGONLxYsX58iRI5w4ccLfoRgTP8JdasrQVz199913/PHHHzz33HOAU77A7okwxphrpdurnkSkvYjsFpF9InLNXSoikkNEvnYtXycipb3Z7vHjx+nRowfdu3dn7ty5REZGAliSMMYYH/BZohCRQGAi0AGoAvQRkSoJVhsKnFbV8sB44M3ktht58SyVK1fmxx9/5PXXX+e3334jKCgotcM3xhjj4sszivrAPlU9oKqRwCygS4J1ugCfuZ7PAVpLMpXKwk//RbVq1diyZQvPPvusVXo1xhgf82VndjEgzG36CNAgqXVUNVpEzgKFgJPuK4nICCCuYHzEqlWrtlsRPwAKk+BYZWF2LK6wY3GFHYsrUnx3ni8TRWJnBgl7zr1ZB1WdAkwBEJGNKe2QyWzsWFxhx+IKOxZX2LG4QkQ2Jr9W4nzZ9HQEKOE2XRz4M6l1RCQbkB/4x4cxGWOMuU6+TBQbgAoiUkZEgoDewA8J1vkBGOh63gNYqhntel1jjMnkfNb05OpzeARYCAQCn6jqDhF5BWeQ7x+A6cDnIrIP50yitxebnuKrmDMgOxZX2LG4wo7FFXYsrkjxschwN9wZY4xJW5mm1pMxxhjfsERhjDHGo3SbKHxV/iMj8uJYjBKRnSKyVUSWiEgpf8SZFpI7Fm7r9RARFZFMe2mkN8dCRHq6Phs7RGRmWseYVrz4P1JSRJaJyGbX/5O0GUM0jYnIJyLyt4hsT2K5iMgHruO0VUTqeLVhVU13D5zO7/1AWSAI2AJUSbDOQ8Ak1/PewNf+jtuPx6IlkMv1/MGsfCxc6+UFVgJrgRB/x+3Hz0UFYDNQwDV9s7/j9uOxmAI86HpeBfifv+P20bFoBtQBtiexvCPwM849bA2Bdd5sN72eUfik/EcGleyxUNVlqnrJNbkW556VzMibzwXAq8BbQHhaBpfGvDkWw4GJqnoaQFX/TuMY04o3x0KBfK7n+bn2nq5MQVVX4vletC7Af9SxFrhJRIomt930migSK/9RLKl1VDUaiCv/kdl4cyzcDcX5xZAZJXssRKQ2UEJVf0zLwPzAm8/F7cDtIrJaRNaKSPs0iy5teXMsXgbuE5EjwALg0bQJLd253u8TIP0OXJRq5T8yAa/fp4jcB4QAzX0akf94PBYiEoBThXhQWgXkR958LrLhND+1wDnL/FVEql2IXLIAAAgZSURBVKnqGR/Hlta8ORZ9gBmq+q6INMK5f6uaqsb6Prx0JUXfm+n1jMLKf1zhzbFARO4EngfuVtWINIotrSV3LPIC1YDlIvI/nDbYHzJph7a3/0fmqWqUqh4EduMkjszGm2MxFJgNoKprgGCcgoFZjVffJwml10Rh5T+uSPZYuJpbJuMkiczaDg3JHAtVPauqhVW1tKqWxumvuVtVU1wMLR3z5v/I9zgXOiAihXGaog6kaZRpw5tjcRhoDSAilXESRVYcu/YHYIDr6qeGwFlVPZbci9Jl05P6rvxHhuPlsXgbyAN84+rPP6yqd/staB/x8lhkCV4ei4VAWxHZCcQAT6nqKf9F7RteHovRwFQRGYnT1DIoM/6wFJGvcJoaC7v6Y14CsgOo/n975x5jVXXF4e8X5O0TRQttdCoPbbV0jNqoaI2PmNT6BzVUqxSRtDVtrBQTmgbbGDWGWFvfL2xIBVMkVDsYwfpAC0MdHj4IzIiiNhajjWlpQqzaqVVc/rHWhTPXO5d7LxcHYX3Jzpyzzj57r73PnbP23uectWw2/nzmXOBvwH+BqTWVuwf2VZIkSdJEdtelpyRJkmQ3IQ1FkiRJUpU0FEmSJElV0lAkSZIkVUlDkSRJklQlDcVehqStktYVUkuVvC29eaGss87l4dlzfbiTOKqBMn4s6ZLYvlTSyMKxOZK+2mQ9n5PUWsM50yUN2dm6a9RvgqSrY/ubktZK+kjSxAbKGiJpvqQuSS9KekbSvk3UdaSkhwr7C8Jb6ZWSrosPRGvqP0kDJK2ID2uTvqCvvR1m+mwT8F4deVvoxQtlnXUuJ7y4ApcBjzSrvCb3TVHPqcDSGs7ZBByyC3TZp4JsZamuuDbjgPuBiQ2UPxO4ubB/FDCw2e2Isr8AvLEz/Yd/DzBpV+iXaccpZxRJaebw1xihrpV0SoU8x0h6NmYhnZLGhPz7Bfm9kvrtoLoVwOg49yx5fIAuuR/9gSG/Qdvja/w2ZNdImhGj5xOA+VHn4JgJnCDpJ5JuLOh8qaQ7GtRzFQVnaZLukfS8PK7DtSGbBowElklaFrJzJK2Kfnyw0ihdUqvcSV+npEWSDgr5ckmzJLUDPys7ZyzwgZn9G8DMNplZJ9Cor6IRwD9KO2b2ipl9EL+FjZLmhX4PlUb8ko6X1C7pBUlPKLyOShot6amYia2VNKpsNvokcGj0/WmS5srjhfToP0k/kHRLoc0/knRz7D4MTGqwrcnO0teWKtNnm/AvdNdFWhSyIcCg2B6Df80KhRkFcAcxosN9/g8GvgIsBvqH/G7gkgp1Lmf7SP3nwELchcKbwNiQ3w9MB4bhPolKH4MeGH+vAWaUl1fcB4bj7qZL8seAUxvUczowq3BsWPztF/nGxf4mto/yD8EN4dDY/wVwdYV6OoHTY/s64NZC/Xf3ct2mAjdVkM+lsRlFK/Av3CBeD4wpXHMDxsf+74EZ+Ne9K4HhIb8Q/wIaYA3wndgehP+eir+dbdvlOpf131A8rkTpOq0Evlbo9819/f+zt6Zc89v76Daz8rX3/sCdsSa/FfcJVM4q4JeSvgS0mdlrks4Cjgeek7sOGYzffCoxX1I3fmO4Al/q+LuZvRrH5wGXA3ficSTmSHoUqNlduJltlvS63IfNa1FHR5Rbj55D8RtTMfrXBZIuw93ejMCD33SWnXtSyDuingF4v21D0gG48WsvtPvBQpaFveg1gib6JjKzdZKOBM4Bzsb75mSgG3jTzDoi6x+AacDjuMPFpdG2fsDbkvYDvmhmi6Lc/wGogdAwZva+pL8A50l6GTcYXXFsq6T/S9rPzN5tuOFJQ6ShSACuBP4JfB1/weFTAX/M7AFJa4BvA09I+iHusniemc2soY5JVnDOJ6li7BBzvz3fwB24fQ/4KXBmHW1ZCFwAbMRnTCa/a9WsJx4h7QbgLuB8SV/GR9UnmtkWSXPxkXM5wp9rXFSHvuW834u8G/eQXDOSLseDFwGca2Y9vISa2XtAG9Am6WPcB9Cf+LTbacPbtsHMTi6rY3+ayxzgKvz63Vd2bCB7djCq3ZZ8RpGA34DeNvfNPxkfLfYgRp+vm9ntuAfKccDTwERJh0aeYao9XvdGoEXS6NifDLTHmv4BZvZnfPmn0ptH7+IuxSvRBkzA4w+URud16WlmHwK/Ak6SexrdH7+BvyPpMOBbveiyGhhfapP8zaIeszMzewfYIum0Yrt706XAy8SznVoxs7vMrDVSDyMhaXzh2cgAfCb0Rhw+PGYX4P34DL4cOLwkl9Rf0jFm9h/gLUkTQj5Q9b0F1uNamtka3A32xcCCgr4H40tPH9ZRdtIk0lAk4Gv2UyStxpedKo1qLwRelLQOOBoPp/gSfkN9UlInsBRfItkhsUQxFfd424U/lJ2N3zSWRHnt+GynnLnA7NLD7LJytwAvAUeY2bMhq1tPM+sGbsKfi6zHY09vwNfsOwpZfwc8JmmZmW3GgyYtiHpW431VzhTgN5GnFX9OsSNWAMfF7AhJJ8q9g34XuFfShhrKKDIKN8xd0bbn8dkEuFGaEvoNA+4xDzE6Efi1pPX4M67SSw+TgWmRfyX+llOtbOu/guyPQEdcyxJn4J5Pkz4gvccmyecESbcBi83sqV1YRwuwxMyO3VV11KDDEuAWM3u6IGsDZprZK32l195MziiS5PPDLPyNoj0SSQdKehV/4aJoJAYAD6eR6DtyRpEkSZJUJWcUSZIkSVXSUCRJkiRVSUORJEmSVCUNRZIkSVKVNBRJkiRJVT4BbM+Bgo+7b1AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_roc_curve(y_test, gs.predict_proba(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'clf__max_depth': 3, 'clf__max_features': 0.5, 'imp__strategy': 'mean'}"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this search we can conclude that the imputation by the 'mean' strategy is generally a slightly better imputation strategy when training a GBRT model on this data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further integrating sklearn and pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helper tool for better sklearn / pandas integration: https://github.com/paulgb/sklearn-pandas by making it possible to embed the feature construction from the raw dataframe directly inside a pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Credits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks to:\n",
    "\n",
    "- Kaggle for setting up the Titanic challenge.\n",
    "\n",
    "- This blog post by Philippe Adjiman for inspiration:\n",
    "\n",
    "http://www.philippeadjiman.com/blog/2013/09/12/a-data-science-exploration-from-the-titanic-in-r/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}