homework6_submit_new1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Home work 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the dataset.  \n",
    "- The dataset was generated using Monte Carlo simluations of the particle collisions that produce other particles and resulting decay products. The main objective is to separate particle-producing collisions from collisions that did not change background source.  \n",
    "- There are 28 features and > 1 million instances.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methods  \n",
    "- It is a classification task with 0 or 1 target response.  \n",
    "    - 1 = particle resulted from collision.  \n",
    "    - 0 = only background signal was received after collision.  \n",
    "- Below is a Tensorflow/Keras implementation of the problem.  \n",
    "- Please note that this training was done on a GPU with weights saved.\n",
    "    - Post-processing was done on a different computer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import keras\n",
    "import numpy as np\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verify that Tensorflow can see the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num GPUs Available:  1\n"
     ]
    }
   ],
   "source": [
    "print(\"Num GPUs Available: \", len(tf.config.list_physical_devices('GPU')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\John Olanipekun\\\\Documents\\\\Machine Learning\\\\SMU_Machine_Learning'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in the dataset.  \n",
    "- We will try if pandas can successfully read in the massive dataset before exploring Dask or PySpark. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th># label</th>\n",
       "      <th>f0</th>\n",
       "      <th>f1</th>\n",
       "      <th>f2</th>\n",
       "      <th>f3</th>\n",
       "      <th>f4</th>\n",
       "      <th>f5</th>\n",
       "      <th>f6</th>\n",
       "      <th>f7</th>\n",
       "      <th>f8</th>\n",
       "      <th>...</th>\n",
       "      <th>f18</th>\n",
       "      <th>f19</th>\n",
       "      <th>f20</th>\n",
       "      <th>f21</th>\n",
       "      <th>f22</th>\n",
       "      <th>f23</th>\n",
       "      <th>f24</th>\n",
       "      <th>f25</th>\n",
       "      <th>f26</th>\n",
       "      <th>mass</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.346368</td>\n",
       "      <td>0.416306</td>\n",
       "      <td>0.999236</td>\n",
       "      <td>0.475342</td>\n",
       "      <td>0.427493</td>\n",
       "      <td>-0.005984</td>\n",
       "      <td>1.989833</td>\n",
       "      <td>0.344530</td>\n",
       "      <td>1.566297</td>\n",
       "      <td>...</td>\n",
       "      <td>4.105282</td>\n",
       "      <td>0.267826</td>\n",
       "      <td>0.378718</td>\n",
       "      <td>1.743123</td>\n",
       "      <td>3.406367</td>\n",
       "      <td>4.350537</td>\n",
       "      <td>-0.352571</td>\n",
       "      <td>1.130032</td>\n",
       "      <td>2.227706</td>\n",
       "      <td>1000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.708236</td>\n",
       "      <td>-0.319394</td>\n",
       "      <td>-1.241873</td>\n",
       "      <td>-0.887231</td>\n",
       "      <td>-0.871906</td>\n",
       "      <td>-0.005984</td>\n",
       "      <td>-0.001047</td>\n",
       "      <td>-1.038225</td>\n",
       "      <td>0.655748</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.178141</td>\n",
       "      <td>-0.877361</td>\n",
       "      <td>-1.483769</td>\n",
       "      <td>-0.573682</td>\n",
       "      <td>-1.693781</td>\n",
       "      <td>-0.545062</td>\n",
       "      <td>-0.299118</td>\n",
       "      <td>-0.662942</td>\n",
       "      <td>-0.193019</td>\n",
       "      <td>750.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.360693</td>\n",
       "      <td>1.794174</td>\n",
       "      <td>0.264738</td>\n",
       "      <td>-0.472273</td>\n",
       "      <td>-0.292344</td>\n",
       "      <td>-1.054221</td>\n",
       "      <td>-1.150495</td>\n",
       "      <td>1.423404</td>\n",
       "      <td>1.270098</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.199511</td>\n",
       "      <td>0.539020</td>\n",
       "      <td>-1.590629</td>\n",
       "      <td>-0.573682</td>\n",
       "      <td>-0.543636</td>\n",
       "      <td>-0.937456</td>\n",
       "      <td>-0.300344</td>\n",
       "      <td>-0.523262</td>\n",
       "      <td>-1.506304</td>\n",
       "      <td>750.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   # label        f0        f1        f2        f3        f4        f5  \\\n",
       "0      1.0 -0.346368  0.416306  0.999236  0.475342  0.427493 -0.005984   \n",
       "1      1.0  1.708236 -0.319394 -1.241873 -0.887231 -0.871906 -0.005984   \n",
       "2      0.0 -0.360693  1.794174  0.264738 -0.472273 -0.292344 -1.054221   \n",
       "\n",
       "         f6        f7        f8  ...       f18       f19       f20       f21  \\\n",
       "0  1.989833  0.344530  1.566297  ...  4.105282  0.267826  0.378718  1.743123   \n",
       "1 -0.001047 -1.038225  0.655748  ... -1.178141 -0.877361 -1.483769 -0.573682   \n",
       "2 -1.150495  1.423404  1.270098  ... -1.199511  0.539020 -1.590629 -0.573682   \n",
       "\n",
       "        f22       f23       f24       f25       f26    mass  \n",
       "0  3.406367  4.350537 -0.352571  1.130032  2.227706  1000.0  \n",
       "1 -1.693781 -0.545062 -0.299118 -0.662942 -0.193019   750.0  \n",
       "2 -0.543636 -0.937456 -0.300344 -0.523262 -1.506304   750.0  \n",
       "\n",
       "[3 rows x 29 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = \"./all_train.csv\"\n",
    "# load the csv file as a dataframe\n",
    "df = pd.read_csv(path)\n",
    "# store the inputs and outputs\n",
    "df.head(n=3)\n",
    "#Train test split\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**EDA done on a different machine showed my that most of the features except 'mass' attribute was skewed.** \n",
    "- So we will scale the 'mass' attribute.  \n",
    "- The target response in this data is the 1st column.  \n",
    "- We will remove the '#' character in the label.  \n",
    "- I also observed and as was confirmed by the metadata, that the dataset does not suffer from class imbalance with respect to the target variable.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split.  \n",
    "- We have a big data so we can set aside validation set and test set.  \n",
    "- so we have 3-way split.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\JOHNOL~1\\AppData\\Local\\Temp/ipykernel_2044/3114481359.py:6: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  train['mass'] = scaler.fit_transform(train['mass'].values.reshape(-1, 1))\n",
      "C:\\Users\\JOHNOL~1\\AppData\\Local\\Temp/ipykernel_2044/3114481359.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  test['mass'] = scaler.transform(test['mass'].values.reshape(-1, 1))\n",
      "C:\\Users\\JOHNOL~1\\AppData\\Local\\Temp/ipykernel_2044/3114481359.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  valid['mass'] = scaler.transform(valid['mass'].values.reshape(-1, 1))\n"
     ]
    }
   ],
   "source": [
    "#train, test = train_test_split(df1, test_size=0.3,random_state=42, stratify = df1['# label'])\n",
    "scaler = StandardScaler()\n",
    "#train, test, y_train_, y_test_ = train_test_split(df, df['# label'], test_size=0.30, random_state=200)\n",
    "train, remain, y_train_, y_remain = train_test_split(df, df['# label'], test_size=0.40, random_state=200)\n",
    "valid, test, y_valid, y_test_ = train_test_split(remain, remain['# label'], test_size=0.50, random_state=200)\n",
    "train['mass'] = scaler.fit_transform(train['mass'].values.reshape(-1, 1))\n",
    "test['mass'] = scaler.transform(test['mass'].values.reshape(-1, 1))  \n",
    "valid['mass'] = scaler.transform(valid['mass'].values.reshape(-1, 1))  \n",
    "train = train.drop(['# label'], axis=1)\n",
    "test = test.drop(['# label'], axis=1)\n",
    "valid = valid.drop(['# label'], axis=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Model architecture, fit and prediction.  \n",
    "### Key hyperparameters.  \n",
    "- Reasonably large epoch size of 100 was chosen but will be checked by early stopping, to prevent overfit.    \n",
    "- Batch size is set to 2048 which is optimally large and is the favourable multiple of 32 bits.  \n",
    "- Call backs:\n",
    "    - I have two callbacks:\n",
    "        1. save the weights of the model training since I will be transferring the notebook to another computer.\n",
    "            - So I can use only the weight for prediction if I lose the current variables.\n",
    "       2. I have **EarlyStopping** criterion to halt training should there be no improvement to metric being monitored.  \n",
    "       - The stopping criterion is set to monitor validation accuracy.  \n",
    "       - This is because validation accuracy is better evaluation of the model performance on unseen test data set.  \n",
    "       - We want to maximize this accuracy so we set the 'mode' parameter to 'max'.  \n",
    "       - **patience** parameter is set to 10 to give the model room to train.  \n",
    "           - This indicates that wait at least 10 epochs before stopping the model training.  \n",
    "           \n",
    "- We set the learning rate to 0.03 arbitrarily.  \n",
    "- Adaptive moment estimation (Adam) is chosen as the optimization algorithm.  \n",
    "- It is quite popular because is one of the algorithms that compute gradient descent fast. \n",
    "       \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "EPOCHS = 100\n",
    "BATCH_SIZE = 2048\n",
    "\n",
    "callbacks =[\n",
    "    keras.callbacks.ModelCheckpoint(\n",
    "        './weights.{epoch:02d}-{val_loss:.2f}.hdf5',\n",
    "        verbose=1), \n",
    "    tf.keras.callbacks.EarlyStopping(\n",
    "    monitor='val_accuracy', \n",
    "    verbose=1,\n",
    "    patience=10,\n",
    "    mode='max',\n",
    "    restore_best_weights=True)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Architecture.  \n",
    "- We will be training on 3 hidden layers and 1 output layer.  \n",
    "- The output layer is set to output 1 value (0 or 1) because we are using binary classification.  \n",
    "- We expect the model to take as input, a minibatch of 2048 instances and 28 attibutes through its input layers and then pass its output to the 1st hidden layer.  \n",
    "- During this forward pass, the algorithm computes output for each neuron in the layer, which is then passed on to the next layer. \n",
    "- At the last layer, it will compute the output error for each training instance, and finds which neuron contributed how much to that error. \n",
    "- It will backpropagate these error back all the way to the 1 layer.  \n",
    "- It then updates the weights of each neuron based on the gradient computed during this revrse process. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss function.  \n",
    "- We choose binary_crossentropy because this task is binary classification.  \n",
    "- Correspondingly we chose the sigmoid activation rather than softmax because we want 0 or 1 value. While softmax is best suited for multiclass outputs.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\John Olanipekun\\AppData\\Roaming\\Python\\Python38\\site-packages\\tensorflow\\python\\keras\\optimizer_v2\\optimizer_v2.py:374: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "2051/2051 [==============================] - 6s 3ms/step - loss: 0.3003 - accuracy: 0.8653 - precision: 0.8460 - recall: 0.8932 - val_loss: 0.2783 - val_accuracy: 0.8740 - val_precision: 0.8606 - val_recall: 0.8929\n",
      "\n",
      "Epoch 00001: saving model to .\\weights.01-0.28.hdf5\n",
      "Epoch 2/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2766 - accuracy: 0.8746 - precision: 0.8566 - recall: 0.8997 - val_loss: 0.2735 - val_accuracy: 0.8762 - val_precision: 0.8614 - val_recall: 0.8969\n",
      "\n",
      "Epoch 00002: saving model to .\\weights.02-0.27.hdf5\n",
      "Epoch 3/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2740 - accuracy: 0.8762 - precision: 0.8593 - recall: 0.8996 - val_loss: 0.2742 - val_accuracy: 0.8771 - val_precision: 0.8658 - val_recall: 0.8926\n",
      "\n",
      "Epoch 00003: saving model to .\\weights.03-0.27.hdf5\n",
      "Epoch 4/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2723 - accuracy: 0.8771 - precision: 0.8609 - recall: 0.8996 - val_loss: 0.2735 - val_accuracy: 0.8766 - val_precision: 0.8726 - val_recall: 0.8820\n",
      "\n",
      "Epoch 00004: saving model to .\\weights.04-0.27.hdf5\n",
      "Epoch 5/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2712 - accuracy: 0.8778 - precision: 0.8624 - recall: 0.8990 - val_loss: 0.2708 - val_accuracy: 0.8777 - val_precision: 0.8562 - val_recall: 0.9079\n",
      "\n",
      "Epoch 00005: saving model to .\\weights.05-0.27.hdf5\n",
      "Epoch 6/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2703 - accuracy: 0.8782 - precision: 0.8624 - recall: 0.9000 - val_loss: 0.2707 - val_accuracy: 0.8783 - val_precision: 0.8650 - val_recall: 0.8966\n",
      "\n",
      "Epoch 00006: saving model to .\\weights.06-0.27.hdf5\n",
      "Epoch 7/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2780 - accuracy: 0.8741 - precision: 0.8566 - recall: 0.8987 - val_loss: 0.2698 - val_accuracy: 0.8788 - val_precision: 0.8702 - val_recall: 0.8905\n",
      "\n",
      "Epoch 00007: saving model to .\\weights.07-0.27.hdf5\n",
      "Epoch 8/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2688 - accuracy: 0.8789 - precision: 0.8633 - recall: 0.9004 - val_loss: 0.2687 - val_accuracy: 0.8795 - val_precision: 0.8651 - val_recall: 0.8993\n",
      "\n",
      "Epoch 00008: saving model to .\\weights.08-0.27.hdf5\n",
      "Epoch 9/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2689 - accuracy: 0.8789 - precision: 0.8637 - recall: 0.8998 - val_loss: 0.2685 - val_accuracy: 0.8794 - val_precision: 0.8537 - val_recall: 0.9160\n",
      "\n",
      "Epoch 00009: saving model to .\\weights.09-0.27.hdf5\n",
      "Epoch 10/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2682 - accuracy: 0.8793 - precision: 0.8638 - recall: 0.9006 - val_loss: 0.2680 - val_accuracy: 0.8796 - val_precision: 0.8741 - val_recall: 0.8871\n",
      "\n",
      "Epoch 00010: saving model to .\\weights.10-0.27.hdf5\n",
      "Epoch 11/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2678 - accuracy: 0.8795 - precision: 0.8639 - recall: 0.9010 - val_loss: 0.2680 - val_accuracy: 0.8802 - val_precision: 0.8612 - val_recall: 0.9066\n",
      "\n",
      "Epoch 00011: saving model to .\\weights.11-0.27.hdf5\n",
      "Epoch 12/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2675 - accuracy: 0.8797 - precision: 0.8641 - recall: 0.9011 - val_loss: 0.2670 - val_accuracy: 0.8799 - val_precision: 0.8635 - val_recall: 0.9027\n",
      "\n",
      "Epoch 00012: saving model to .\\weights.12-0.27.hdf5\n",
      "Epoch 13/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2674 - accuracy: 0.8798 - precision: 0.8643 - recall: 0.9011 - val_loss: 0.2690 - val_accuracy: 0.8799 - val_precision: 0.8568 - val_recall: 0.9125\n",
      "\n",
      "Epoch 00013: saving model to .\\weights.13-0.27.hdf5\n",
      "Epoch 14/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2671 - accuracy: 0.8799 - precision: 0.8643 - recall: 0.9013 - val_loss: 0.2662 - val_accuracy: 0.8801 - val_precision: 0.8606 - val_recall: 0.9074\n",
      "\n",
      "Epoch 00014: saving model to .\\weights.14-0.27.hdf5\n",
      "Epoch 15/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2670 - accuracy: 0.8799 - precision: 0.8646 - recall: 0.9010 - val_loss: 0.2680 - val_accuracy: 0.8800 - val_precision: 0.8599 - val_recall: 0.9081\n",
      "\n",
      "Epoch 00015: saving model to .\\weights.15-0.27.hdf5\n",
      "Epoch 16/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2670 - accuracy: 0.8800 - precision: 0.8645 - recall: 0.9012 - val_loss: 0.2661 - val_accuracy: 0.8804 - val_precision: 0.8666 - val_recall: 0.8994\n",
      "\n",
      "Epoch 00016: saving model to .\\weights.16-0.27.hdf5\n",
      "Epoch 17/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2667 - accuracy: 0.8803 - precision: 0.8646 - recall: 0.9018 - val_loss: 0.2693 - val_accuracy: 0.8798 - val_precision: 0.8629 - val_recall: 0.9033\n",
      "\n",
      "Epoch 00017: saving model to .\\weights.17-0.27.hdf5\n",
      "Epoch 18/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2669 - accuracy: 0.8802 - precision: 0.8651 - recall: 0.9008 - val_loss: 0.2667 - val_accuracy: 0.8800 - val_precision: 0.8573 - val_recall: 0.9118\n",
      "\n",
      "Epoch 00018: saving model to .\\weights.18-0.27.hdf5\n",
      "Epoch 19/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2668 - accuracy: 0.8800 - precision: 0.8643 - recall: 0.9015 - val_loss: 0.2658 - val_accuracy: 0.8806 - val_precision: 0.8619 - val_recall: 0.9067\n",
      "\n",
      "Epoch 00019: saving model to .\\weights.19-0.27.hdf5\n",
      "Epoch 20/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2665 - accuracy: 0.8803 - precision: 0.8650 - recall: 0.9013 - val_loss: 0.2651 - val_accuracy: 0.8809 - val_precision: 0.8657 - val_recall: 0.9018\n",
      "\n",
      "Epoch 00020: saving model to .\\weights.20-0.27.hdf5\n",
      "Epoch 21/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2663 - accuracy: 0.8805 - precision: 0.8651 - recall: 0.9015 - val_loss: 0.2654 - val_accuracy: 0.8811 - val_precision: 0.8605 - val_recall: 0.9096\n",
      "\n",
      "Epoch 00021: saving model to .\\weights.21-0.27.hdf5\n",
      "Epoch 22/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2659 - accuracy: 0.8808 - precision: 0.8651 - recall: 0.9022 - val_loss: 0.2687 - val_accuracy: 0.8812 - val_precision: 0.8658 - val_recall: 0.9023\n",
      "\n",
      "Epoch 00022: saving model to .\\weights.22-0.27.hdf5\n",
      "Epoch 23/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2659 - accuracy: 0.8807 - precision: 0.8650 - recall: 0.9022 - val_loss: 0.2654 - val_accuracy: 0.8809 - val_precision: 0.8686 - val_recall: 0.8977\n",
      "\n",
      "Epoch 00023: saving model to .\\weights.23-0.27.hdf5\n",
      "Epoch 24/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2658 - accuracy: 0.8807 - precision: 0.8647 - recall: 0.9026 - val_loss: 0.2648 - val_accuracy: 0.8807 - val_precision: 0.8648 - val_recall: 0.9026\n",
      "\n",
      "Epoch 00024: saving model to .\\weights.24-0.26.hdf5\n",
      "Epoch 25/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2655 - accuracy: 0.8810 - precision: 0.8652 - recall: 0.9025 - val_loss: 0.2656 - val_accuracy: 0.8806 - val_precision: 0.8536 - val_recall: 0.9189\n",
      "\n",
      "Epoch 00025: saving model to .\\weights.25-0.27.hdf5\n",
      "Epoch 26/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2655 - accuracy: 0.8810 - precision: 0.8654 - recall: 0.9023 - val_loss: 0.2654 - val_accuracy: 0.8814 - val_precision: 0.8635 - val_recall: 0.9063\n",
      "\n",
      "Epoch 00026: saving model to .\\weights.26-0.27.hdf5\n",
      "Epoch 27/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2654 - accuracy: 0.8810 - precision: 0.8651 - recall: 0.9026 - val_loss: 0.2668 - val_accuracy: 0.8809 - val_precision: 0.8563 - val_recall: 0.9155\n",
      "\n",
      "Epoch 00027: saving model to .\\weights.27-0.27.hdf5\n",
      "Epoch 28/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2652 - accuracy: 0.8811 - precision: 0.8654 - recall: 0.9024 - val_loss: 0.2650 - val_accuracy: 0.8806 - val_precision: 0.8728 - val_recall: 0.8912\n",
      "\n",
      "Epoch 00028: saving model to .\\weights.28-0.27.hdf5\n",
      "Epoch 29/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2652 - accuracy: 0.8810 - precision: 0.8656 - recall: 0.9021 - val_loss: 0.2714 - val_accuracy: 0.8803 - val_precision: 0.8764 - val_recall: 0.8854\n",
      "\n",
      "Epoch 00029: saving model to .\\weights.29-0.27.hdf5\n",
      "Epoch 30/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2651 - accuracy: 0.8810 - precision: 0.8655 - recall: 0.9023 - val_loss: 0.2678 - val_accuracy: 0.8809 - val_precision: 0.8617 - val_recall: 0.9075\n",
      "\n",
      "Epoch 00030: saving model to .\\weights.30-0.27.hdf5\n",
      "Epoch 31/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2651 - accuracy: 0.8810 - precision: 0.8652 - recall: 0.9027 - val_loss: 0.2641 - val_accuracy: 0.8815 - val_precision: 0.8617 - val_recall: 0.9088\n",
      "\n",
      "Epoch 00031: saving model to .\\weights.31-0.26.hdf5\n",
      "Epoch 32/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2649 - accuracy: 0.8811 - precision: 0.8652 - recall: 0.9029 - val_loss: 0.2645 - val_accuracy: 0.8809 - val_precision: 0.8613 - val_recall: 0.9082\n",
      "\n",
      "Epoch 00032: saving model to .\\weights.32-0.26.hdf5\n",
      "Epoch 33/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2652 - accuracy: 0.8809 - precision: 0.8655 - recall: 0.9021 - val_loss: 0.2667 - val_accuracy: 0.8800 - val_precision: 0.8553 - val_recall: 0.9149\n",
      "\n",
      "Epoch 00033: saving model to .\\weights.33-0.27.hdf5\n",
      "Epoch 34/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2651 - accuracy: 0.8812 - precision: 0.8656 - recall: 0.9024 - val_loss: 0.2677 - val_accuracy: 0.8803 - val_precision: 0.8776 - val_recall: 0.8840\n",
      "\n",
      "Epoch 00034: saving model to .\\weights.34-0.27.hdf5\n",
      "Epoch 35/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2651 - accuracy: 0.8812 - precision: 0.8657 - recall: 0.9023 - val_loss: 0.2645 - val_accuracy: 0.8812 - val_precision: 0.8672 - val_recall: 0.9004\n",
      "\n",
      "Epoch 00035: saving model to .\\weights.35-0.26.hdf5\n",
      "Epoch 36/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2647 - accuracy: 0.8812 - precision: 0.8653 - recall: 0.9030 - val_loss: 0.2656 - val_accuracy: 0.8803 - val_precision: 0.8649 - val_recall: 0.9015\n",
      "\n",
      "Epoch 00036: saving model to .\\weights.36-0.27.hdf5\n",
      "Epoch 37/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2647 - accuracy: 0.8812 - precision: 0.8656 - recall: 0.9026 - val_loss: 0.2669 - val_accuracy: 0.8812 - val_precision: 0.8581 - val_recall: 0.9136\n",
      "\n",
      "Epoch 00037: saving model to .\\weights.37-0.27.hdf5\n",
      "Epoch 38/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2649 - accuracy: 0.8813 - precision: 0.8658 - recall: 0.9025 - val_loss: 0.2659 - val_accuracy: 0.8810 - val_precision: 0.8553 - val_recall: 0.9173\n",
      "\n",
      "Epoch 00038: saving model to .\\weights.38-0.27.hdf5\n",
      "Epoch 39/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2648 - accuracy: 0.8812 - precision: 0.8656 - recall: 0.9025 - val_loss: 0.2638 - val_accuracy: 0.8813 - val_precision: 0.8663 - val_recall: 0.9018\n",
      "\n",
      "Epoch 00039: saving model to .\\weights.39-0.26.hdf5\n",
      "Epoch 40/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2648 - accuracy: 0.8812 - precision: 0.8656 - recall: 0.9025 - val_loss: 0.2695 - val_accuracy: 0.8803 - val_precision: 0.8517 - val_recall: 0.9211\n",
      "\n",
      "Epoch 00040: saving model to .\\weights.40-0.27.hdf5\n",
      "Epoch 41/100\n",
      "2051/2051 [==============================] - 5s 3ms/step - loss: 0.2648 - accuracy: 0.8812 - precision: 0.8658 - recall: 0.9024 - val_loss: 0.2693 - val_accuracy: 0.8804 - val_precision: 0.8793 - val_recall: 0.8818\n",
      "\n",
      "Epoch 00041: saving model to .\\weights.41-0.27.hdf5\n",
      "Restoring model weights from the end of the best epoch.\n",
      "Epoch 00041: early stopping\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Dense(128, activation='relu'),\n",
    "    tf.keras.layers.Dense(256, activation='relu'),\n",
    "    tf.keras.layers.Dense(256, activation='relu'),\n",
    "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "model.compile(\n",
    "    loss=tf.keras.losses.binary_crossentropy,\n",
    "    optimizer=tf.keras.optimizers.Adam(lr=0.03),\n",
    "    metrics=[\n",
    "        tf.keras.metrics.BinaryAccuracy(name='accuracy'),\n",
    "        tf.keras.metrics.Precision(name='precision'),\n",
    "        tf.keras.metrics.Recall(name='recall')\n",
    "    ]\n",
    ")\n",
    "\n",
    "history = model.fit(train.values,\n",
    "                    y_train_.values,\n",
    "                    batch_size=BATCH_SIZE,\n",
    "                    epochs=EPOCHS, \n",
    "                    validation_data=(valid.values, y_valid.values), callbacks=callbacks\n",
    "                   )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training history.  \n",
    "- We have 100 epochs.  \n",
    "- We want to minimize our loss function (binary cross entropy).  \n",
    "- We want to maximize our accuracy, precision and recall for the validation set.  \n",
    "    - These start from >80% and are increasing up to 41st epoch when the early stopping criterion condition halted the training because there was no significant improvement to validation accuracy after 10 epochs.  \n",
    "-  **Validation accuracy = 88%, precision=88%, recall = 88%**.  \n",
    "- We will see the performance of the model weights on the test set.\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prediction.  \n",
    "- We predict on the test set with decision threshld set to 0.5. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(test.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction_classes = [\n",
    "    1 if prob > 0.5 else 0 for prob in np.ravel(predictions)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation.  \n",
    "- The model produced very similar Accuracy, precision and recall values (88%, 86%, 91% respectfully) as we saw with the validation set during training. \n",
    "- Therefore, we can see that the model will be accurate in prediction if particle collision will form another particle or just background signal 88 % of the time.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[597304 101952]\n",
      " [ 63924 636820]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "print(confusion_matrix(y_test_, prediction_classes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.88\n",
      "Precision: 0.86\n",
      "Recall: 0.91\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score\n",
    "\n",
    "print(f'Accuracy: {accuracy_score(y_test_, prediction_classes):.2f}')\n",
    "print(f'Precision: {precision_score(y_test_, prediction_classes):.2f}')\n",
    "print(f'Recall: {recall_score(y_test_, prediction_classes):.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backup the history.  \n",
    "-We store the training loss, accuracy, precision as history."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss=history.history['loss']\n",
    "accuracy=history.history['accuracy']\n",
    "precision=history.history['precision']\n",
    "recall= history.history['recall']\n",
    "hist_dict={}\n",
    "hist_dict['loss']=loss\n",
    "hist_dict['accuracy']=accuracy\n",
    "hist_dict['precision']=precision\n",
    "hist_dict['recall']=recall\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'loss': [0.3002937138080597,\n",
       "  0.2766357660293579,\n",
       "  0.2739705443382263,\n",
       "  0.272262841463089,\n",
       "  0.2711673378944397,\n",
       "  0.2703096270561218,\n",
       "  0.27799054980278015,\n",
       "  0.2687784731388092,\n",
       "  0.26892217993736267,\n",
       "  0.26824745535850525,\n",
       "  0.2678259313106537,\n",
       "  0.26747849583625793,\n",
       "  0.26744091510772705,\n",
       "  0.2671263813972473,\n",
       "  0.2669764757156372,\n",
       "  0.2669505476951599,\n",
       "  0.26666656136512756,\n",
       "  0.2669451832771301,\n",
       "  0.26684674620628357,\n",
       "  0.26654812693595886,\n",
       "  0.26627352833747864,\n",
       "  0.26587343215942383,\n",
       "  0.2659132778644562,\n",
       "  0.2657637894153595,\n",
       "  0.2655166983604431,\n",
       "  0.2654632329940796,\n",
       "  0.2654375731945038,\n",
       "  0.26519742608070374,\n",
       "  0.2651599943637848,\n",
       "  0.26513949036598206,\n",
       "  0.2651105523109436,\n",
       "  0.2649342119693756,\n",
       "  0.265164852142334,\n",
       "  0.26505717635154724,\n",
       "  0.2651055157184601,\n",
       "  0.26471075415611267,\n",
       "  0.26474684476852417,\n",
       "  0.26493045687675476,\n",
       "  0.2648445665836334,\n",
       "  0.2647569477558136,\n",
       "  0.26484259963035583],\n",
       " 'accuracy': [0.865333080291748,\n",
       "  0.8745778799057007,\n",
       "  0.8761605024337769,\n",
       "  0.8771002292633057,\n",
       "  0.8778059482574463,\n",
       "  0.8781793117523193,\n",
       "  0.8741178512573242,\n",
       "  0.8789352178573608,\n",
       "  0.8789302110671997,\n",
       "  0.8792778849601746,\n",
       "  0.8795202374458313,\n",
       "  0.8797131180763245,\n",
       "  0.8797900080680847,\n",
       "  0.8798807263374329,\n",
       "  0.8799468874931335,\n",
       "  0.879999041557312,\n",
       "  0.8802750110626221,\n",
       "  0.8802078366279602,\n",
       "  0.8799802660942078,\n",
       "  0.8803368806838989,\n",
       "  0.8804616928100586,\n",
       "  0.8807899951934814,\n",
       "  0.8806873559951782,\n",
       "  0.8807119131088257,\n",
       "  0.8809573650360107,\n",
       "  0.8810240626335144,\n",
       "  0.8809630870819092,\n",
       "  0.8810528516769409,\n",
       "  0.8810368776321411,\n",
       "  0.8810328841209412,\n",
       "  0.8810116648674011,\n",
       "  0.8810988068580627,\n",
       "  0.8809359669685364,\n",
       "  0.8811666369438171,\n",
       "  0.8811600208282471,\n",
       "  0.8812268972396851,\n",
       "  0.8812447786331177,\n",
       "  0.8813128471374512,\n",
       "  0.8811802268028259,\n",
       "  0.881208598613739,\n",
       "  0.8812414407730103],\n",
       " 'precision': [0.8460310697555542,\n",
       "  0.8566490411758423,\n",
       "  0.8593307137489319,\n",
       "  0.8608666062355042,\n",
       "  0.8624094724655151,\n",
       "  0.8623570799827576,\n",
       "  0.8565852046012878,\n",
       "  0.8633193969726562,\n",
       "  0.8637336492538452,\n",
       "  0.863753080368042,\n",
       "  0.8638542890548706,\n",
       "  0.8641332983970642,\n",
       "  0.8642610907554626,\n",
       "  0.8642637729644775,\n",
       "  0.8646090030670166,\n",
       "  0.864542543888092,\n",
       "  0.8645913004875183,\n",
       "  0.865136444568634,\n",
       "  0.8642504215240479,\n",
       "  0.8650239706039429,\n",
       "  0.8651126623153687,\n",
       "  0.8651042580604553,\n",
       "  0.8649510145187378,\n",
       "  0.8647069931030273,\n",
       "  0.8651942014694214,\n",
       "  0.8654346466064453,\n",
       "  0.8651343584060669,\n",
       "  0.8654287457466125,\n",
       "  0.8656190037727356,\n",
       "  0.86545729637146,\n",
       "  0.8651835918426514,\n",
       "  0.8651804327964783,\n",
       "  0.8654546141624451,\n",
       "  0.86564040184021,\n",
       "  0.8656520843505859,\n",
       "  0.8653321862220764,\n",
       "  0.8656355142593384,\n",
       "  0.8657970428466797,\n",
       "  0.8655751347541809,\n",
       "  0.8656418919563293,\n",
       "  0.8657810091972351],\n",
       " 'recall': [0.893197774887085,\n",
       "  0.899689257144928,\n",
       "  0.8995554447174072,\n",
       "  0.8995697498321533,\n",
       "  0.8990249037742615,\n",
       "  0.8999888300895691,\n",
       "  0.8986782431602478,\n",
       "  0.9004031419754028,\n",
       "  0.8997973799705505,\n",
       "  0.9005950689315796,\n",
       "  0.9010255336761475,\n",
       "  0.9010836482048035,\n",
       "  0.9010831713676453,\n",
       "  0.9012946486473083,\n",
       "  0.9009579420089722,\n",
       "  0.9011765122413635,\n",
       "  0.9017613530158997,\n",
       "  0.9008236527442932,\n",
       "  0.9015498757362366,\n",
       "  0.9012898802757263,\n",
       "  0.9014589190483093,\n",
       "  0.9022489786148071,\n",
       "  0.9022247195243835,\n",
       "  0.9026318788528442,\n",
       "  0.9025170803070068,\n",
       "  0.9023318290710449,\n",
       "  0.9026161432266235,\n",
       "  0.9024085402488708,\n",
       "  0.9020994305610657,\n",
       "  0.9023204445838928,\n",
       "  0.9026609063148499,\n",
       "  0.9028719067573547,\n",
       "  0.9020946621894836,\n",
       "  0.902376115322113,\n",
       "  0.90234375,\n",
       "  0.9029585719108582,\n",
       "  0.9025680422782898,\n",
       "  0.9024990200996399,\n",
       "  0.9025014042854309,\n",
       "  0.9024732708930969,\n",
       "  0.9023528099060059]}"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_dict_df = pd.DataFrame.from_dict(hist_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_dict_df.to_csv('hist.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\olani\\\\OneDrive\\\\Documents\\\\Data Science\\\\SMU-Data Science\\\\Quantifying the world'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.300294</td>\n",
       "      <td>0.865333</td>\n",
       "      <td>0.846031</td>\n",
       "      <td>0.893198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.276636</td>\n",
       "      <td>0.874578</td>\n",
       "      <td>0.856649</td>\n",
       "      <td>0.899689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.273971</td>\n",
       "      <td>0.876161</td>\n",
       "      <td>0.859331</td>\n",
       "      <td>0.899555</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       loss  accuracy  precision    recall\n",
       "0  0.300294  0.865333   0.846031  0.893198\n",
       "1  0.276636  0.874578   0.856649  0.899689\n",
       "2  0.273971  0.876161   0.859331  0.899555"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "path = r\"./hist.csv\"\n",
    "# load the csv file as a dataframe\n",
    "history = pd.read_csv(path)\n",
    "# store the inputs and outputs\n",
    "history.head(n=3)\n",
    "#Train test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'epoch')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA5n0lEQVR4nO3deXxddZn48c9zb272vUnTpFvShS4gUChlRxShBUHcZoYqqLgA/kBhRhlklp/MOI7OjI76ExQXRAUGBIUREClQFWSpdKe0pVCatmnTNkmTNOvN3Z7fH9+T9Ca9SW7b3NyQPO/X67zOfs5zT5vvc873exZRVYwxxpiBfOkOwBhjzNhkCcIYY0xCliCMMcYkZAnCGGNMQpYgjDHGJGQJwhhjTEKWIEzaiMjvReST6Y4j3UTkQhHZk+Syd4jI/amOyRiAjHQHYN5ZRKQjbjQX6AGi3vj1qvpAsttS1UtHMrbhiEg1UAsEVDUymvs25p3IEoQ5Kqqa3zssIjuBz6rqcwOXE5EMK4QnFvs3H3+sismMiN5qEhG5TUT2A/eKSImIPCkijSLS4g1Pi1vnTyLyWW/4UyLyooh8y1u2VkQGvcLw9rNXRNpFZJuIXORN94nIV0TkbRE5KCIPi0ipt9oLXr9VRDpE5Oyj+H13iMgjInK/t89NInKCiNwuIg0iUicil8QtXyUij4tIs4hsF5HPxc3LEZGfe79zC3DGgH1VichvvONWKyJfTDLG4Y53qYjcKyL13vz/jZt3pYhsEJE279gt86bvFJH3DTgO93vD1SKiIvIZEdkN/MGb/oiI7BeRQyLygoicOOC3f1tEdnnzX/Sm/U5EvjDg97wmIh9M5reb1LAEYUbSFKAUmAlch/v/da83PgPoBu4cYv0zgW1AGfCfwD0iIgMXEpF5wE3AGapaACwFdnqzvwh8EHg3UAW0AHd58y7w+sWqmq+qrxzl77sCuA8oAdYDK7zfOBX4V+BHccs+COzxYvgo8O+9SQz4KjDb65YCfe0wIuIDngA2etu9CLhFRJYmEd9wx/s+XLXgicBk4DvePpcAvwRuBYpxx2lnEvvr9W5ggfdbAH4PzPX2sQ6Ir3b8FnA6cA7u/8rfAzHgF8DVvQuJyCm43//UUcRhRpqqWmfdMXW4QuR93vCFQAjIHmL5U4GWuPE/4aqoAD4FbI+blwsoMCXBduYADcD7cO0J8fO2AhfFjVcCYVx1arW3zYxj+K13AM/GjV8BdAB+b7zA23YxMB3XLlMQt/w3gJ97wzuAZXHzrgP2eMNnArsH7Pt24N64OO5PMua+4+0dhxhQkmC5HwHfGe7feOD+447nrCFiKPaWKcIlsG7glATLZQHNwFxv/FvAD9L9f3yid3YFYUZSo6oGe0dEJFdEfuRVJ7ThqniKRcQ/yPr7ewdUtcsbzB+4kKpuB27BFVYNIvKQiFR5s2cCj4lIq4i04hJGFKg4rl/mHIgb7gaaVDUaN94bbxXQrKrtccvvwp0R482vGzCv10ygqjd+7zf8QzLxD3O8p3sxtSRYdTrw9nDbH0LfbxERv4h806umauPwlUiZ12Un2peq9gAPA1d7V1HLcVc8Jo0sQZiRNPDVwF8C5gFnqmohh6t4jqg2Ouodqf6Pqp6HK1AV+A9vVh1wqaoWx3XZqro3QXypUg+UikhB3LQZwF5veB+uUI6f16sOqB0Qf4GqXpbEfoc63nVeTMUJ1qvDVXcl0om7mus1JcEy8cf1Y8CVuKu7ItxVRm8MTUBwiH39Avg4rlqtS4++CtCMMEsQJpUKcGfWrV5D8VdHYqMiMk9E3isiWbgCp5vDt9reDXxdRGZ6y5aLyJXevEZcNcuskYhjMKpaB7wMfENEskXkZOAzHK6Lfxi43WtUngbEN86+CrR5jfA53hn5SSLSryF7EIMeb1Xdh2sb+IG334CI9CaQe4BrReQicY38U0VkvjdvA3CVt/xiXHvKcDH0AAdxieXf42KIAT8D/ttriPeLyNnevyNeQogB38auHsYESxAmlb4L5ODOHFcBT4/QdrOAb3rb3Y9rDP0Hb973gMeBZ0Sk3dvvmdBXbfV14CWv+uasEYonkeW4s+d64DHgq6r6rDfvX3DVSrXAM8QVhl6V1RW49oNa3G/8Ke5sfDjfZejjfQ2uPeYNXBvOLd4+XwWuxTVaHwKex12ZAfwz7oy/xYv7f4aJ4Zfeb9sLbPHiiPdlYBOwGtfm8B/0L4d+CbwLsIcBxwBRtQ8GGWPGBhH5BHCdV31o0syuIIwxY4KI5AL/B/hxumMxjiUIM2GJyN3iHpgb2N2d7tgmGu85j0bcnWLDVWOZUWJVTMYYYxKyKwhjjDEJjauX9ZWVlWl1dXW6wzDGmHeMtWvXNqlqeaJ54ypBVFdXs2bNmnSHYYwx7xgismuweVbFZIwxJiFLEMYYYxKyBGGMMSYhSxDGGGMSsgRhjDEmIUsQxhhjErIEYYwxJqFx9RyEMWb8isQiNAebaQm2UFNUQ6Y/c9T23dTdxCv1r9AcbGZu8VxOKD2BSdmTSPDJ9HHFEoQx70AxjVHXXsebLW8SjUXJDeSSH8gnL5DXrzuWQrQj1MHejr0c6DpAXiCPspwyynLKyAvkpeCXOKpKc7CZ2kO11LbVUt9RT1N3E43djTR1NdHU3URzsBn1Pl5XECjg4uqLeX/N+1k8ZTE+GdnKkHAszMaGjbxU/xIv7X2Jrc1bj1imJKuEE0pOYG7J3L5+cVYxWf4sMv2ZZPmzyPJnvaOTyLh6Wd/ixYvVnqQ27zThaJioRvGLH5/48ImvX6HSHenmrZa3eKP5Dd5sebOv3x3pHmKrTqYvk5LsEkqzS/u6+PFwLMyejj3sbd/L3g7Xtfa0JtxWTkZOX7Ioyyljcu5kphdMZ0bBDGYUzqAqv4qALzBoLNFYlOZgMw3dDezv3M/OQzv7EsLOQztpC7X1LesXP5OyJ1GWW0Z5Tnm//eZn5vPy3pdZuXslXZEuJudO5rKay3j/rPczr2TeEQVye6id+o569nTsob6jnmAkiN/nxy9e5w1n+DLojnSzev9qVu1bRWe4kwzJ4JTJp3De1PM4t+pcpuRNYXvrdt5sedN1zW+yvXU7wWhw4M/t92+Q5c8i4O9/bAaWvSKCT3x9cfnEdzhOn5+8jDzyMvPID+T3nQz09kuyS7hi9hWDxjAUEVmrqosTzrMEYczoC0aC/KnuTzy540le2vsSEY30my9IXyERjoX7nTmfUHoC80vnM69kHieUnEB2RjYd4Q46w510hbv6DbeH2mnpaaE52Exzd7PrB5v7FWgZvgyq8qqYVjCNqflTXVcwlSm5U+iOdNPU3ZSw29+5n65IV992/OKnMq+SGYUzmF4wnZjGaOxqpKG7gcauRg4GDxLTWL/fWZ5TTnVRNTWFNdQU1bjhohqm5E7B7/MPeQy7I938qe5PPLXjKV7c+yIRjTCraBZnVp5JU3cTe9r3sLdjb7/Ek4zKvEqXEKaey5lTziQ/M3/I5aOxKHs69rC9ZTsd4Q56oj2EoiGC0SChaKhvPBQNDXk1oapENUpMY0Q16oZjbjgSi9AV6aIz3On+fUOu33v8J+dMZuVfrzyq39nLEoSZ8A50HmBbyzYWTV5EQWZBWmKIxqKsObCGJ95+gud2P0dnuJPJuZNZWr2U0uzSvoIhprG+LqpRcjJymFcyj3ml86jKqxqRKouucBfNwWYyfBmU55QPWxgn0lstVNdex+723exq20Vdmxve3b6bgC9AeU455bnlTM6d7Ia98YrcCmYUzhixf4vWYCvP7HqG3+34HVubt1KRW8HUgqlMy++f9KbmTSUvkEdEI8Q0RiQWcYVxzBXIgjA5d/I7plooGovSFekiGAlSnpvwfXvDsgRhJpyOUAdrDqxh1b5VvFL/CjsO7QAgNyOXD8z+AMsXLGdW0axhtxONRVnfsJ7n9zyPiFCVV0VlXiVT8qZQlV81aAEXjARp7WmlJdhCS7CFVftW8bva39HQ1UBeII+LZ17MFbOu4PSK04+pcDZmpAyVIKyR2oy6rnAXB7sP0tjd6BohvSqLxq5GmoPNTMqZRHVhdV/Vw/SC6UfU3/aKaYzWnlZXldHVwKamTazat4rXGl8jqlGy/dmcXnE6H5rzIeaUzOH3tb/nN2/9hoe2PcQ5Vefw8QUf57yp5/Vr5IzGoqxrWMeKnStYuXslTd1NfXXr4Vi43/7zA/lU5ldSklVCe6id1p5WWntaj2gfyJAMzpt6HreecSsXTruQ7IzsET6qxow8u4Iww4ppjN1tu9navJWtB7eypXkLrcFWQjFXrxqOhgnFQoRjYTc+oBBNhl/8TMqZRGl2aV/CiJ83rWAa1YXVlOeW09zdfDi5dDX1q7/3iY8TJ53IWZVncVblWZw6+dQj7uQ52H2QX7/5ax7e9jAN3Q1ML5jO8vnLmVM8h5W7V/Lcruc4GDxItj+b86edzyUzL+GCaReQnZFNc7CZ+o569nXuY3/n/r7h1p5WCjMLKcoqoiSrhOLsYoqzivuGZxXNoiS75Nj/EYxJkbRVMYnIMuB7gB/4qap+c8D8EuBnwGwgCHxaVV9PZt1ELEGMjIPdB3m5/mW2HNzC1uatvNH8Bp3hTgACvgBzS+ZSkVtBwBcg059Jpj+zbzjgCxDwBYasw83yZ1GW0//ulJLskn5n8e2hdna17XJ3uRyqZWebu+OlOdhMaXZpX132wP6solkUZRUl9TvDsTArd63kga0PsKFxA+Du1Dl/6vlcUn0J5089n9xA7rEfSGPeAdKSIETED7wJXAzsAVYDy1V1S9wy/wV0qOq/iMh84C5VvSiZdROxBOGoKk/VPsVbLW9xZuWZnFZxGln+rCHX6b2r5vG3H+fl+pf7qmdOKD2BhaULWThpIQsmLWB20exBq3veybYc3EJDVwNLpiyxpGAmlHS1QSwBtqvqDi+Ih4ArgfhCfiHwDQBVfUNEqkWkApiVxLrjXkuwhdxA7rCFe7yeaA9fX/V1Htv+GIJwz+v3kO3PZvGUxZxbdS7nVJ1DTVENIkJMY6w9sJYn3n6CZ3c9S0e4g4rcCj554idZVr2MuSVzyfBNjGaqhZNcEjTGHJbKv/6pQF3c+B7gzAHLbAQ+DLwoIkuAmcC0JNcdFarKvs59ZPozKcgsOKrC+lj3t/bAWu7bch9/rPsjFXkV3Lr4Vi6eefGwt97Vd9Tzt3/6W7Yc3ML1J1/Pp078FOsa1vHS3pd4uf5l/mP1fwAwJW8KiyYvYkPDBvZ17iM3I9fdVTP7ChZXLLa7aowxQGoTRKLSbGB91jeB74nIBmATsB6IJLmu24nIdcB1ADNmzDjWWAf1zK5n+PLzX+4bz/Rlkp+ZT2FmIfmB/L6kEcPdt64oqq6LEUMQ5pXM44wpZ7CoYhGFmYUJ9xOOhnl659Pct+U+tjZvpSiriE8s/ASr9q3iS89/ibMqz+L2M28f9NbMV+pf4e9f+HsisQjff+/3uXD6hQBcMO0CLph2AQB7O/bycv3LvLz3ZV7d9yrzJ83n5tNu5r0z3ktORs7IHjhjzDteKtsgzgbuUNWl3vjtAKr6jUGWF6AWOBk48WjW7ZWKNoibVt7E1oNb+dzJn6Mj3EFbqI32UDsdoQ7aQ+20hdoIx8II7jH5vr732Hw4GmZby7a+ZeaXzueMKWdwxpQzOK3iNKKxKI+8+QgPvfEQjd2N1BTVcPWCq7li9hXkZOQQiUX41bZfcdf6u+iOdvOJhZ/g+pOv76snV1Xuef0evr/++8wqmsV33/NdZhbOHNFjYIwZv9LVSJ2Ba2i+CNiLa2j+mKpujlumGOhS1ZCIfA44X1U/kcy6iYx0gmgLtfHuX72bj83/GLeecesxbycYCbKpaROr969mzYE1bGzYSCgWQhAyfBmEY2HOqTqHaxZewzlV5yR88VhTdxPfXftdfvv2b6nIreDLZ3yZ86rO459f+mee2/0cl1Zfyh3n3GENrMaYo5KWRmpVjYjITcAK3K2qP1PVzSJygzf/bmAB8EsRieIaoD8z1LqpinUwf9z9RyKxCMuqlx3XdrIzsvuuGsA1JL/W+BprDqyhPdTOh+d8mDklc4bcRllOGf923r/x0RM+ytf/8nVuff5W8gJ5BCNBbl18K9csvOYd83oAY8w7gz0oN4TPP/d5drTu4OmPPD2mCt/eaqkndjzBLafd0pd4jDHmaNmrNo7BoZ5DrKpfNSbPzP0+P1fNv4qr5l+V7lCMMeOYfXJ0ECt3rySiEZbWLE13KMYYkxaWIAaxYucKpuVPY2GpPTxljJmYLEEk0BJs4S/7/sKymmVjrnrJGGNGiyWIBJ7b/RxRjbK02qqXjDETlyWIBFbsXMHMwpnMK5mX7lCMMSZtLEEM0NTdxOr9q1lavdSql4wxE5oliAFW7lpJTGNWvWSMmfAsQQzw9M6nmVU0i7nFc9MdijHGpJUliDiNXY2sPbDWqpeMMQZLEP08u+tZFLXqJWOMwRJEPyt2rmBO8RxmF89OdyjGGJN2liA8BzoPsK5hnV09GGOMxxKE55ldzwBYgjDGGI8lCM+KnSuYVzKPmqKadIdijDFjgiUIYF/HPjY2brSrB2OMiWMJAqteMsaYRCxB4KqXFpQuYEbhjHSHYowxY8aETxBd4S66I90sqzm+704bY8x4M+E/OZobyOWxKx8jEoukOxRjjBlTJvwVRK8M34TPlcYY048lCGOMMQlZgjDGGJOQJQhjjDEJWYIwxhiTkCUIY4wxCaU0QYjIMhHZJiLbReQrCeYXicgTIrJRRDaLyLVx824Wkde96bekMk5jjDFHSlmCEBE/cBdwKbAQWC4iCwcsdiOwRVVPAS4Evi0imSJyEvA5YAlwCnC5iNg3QI0xZhSl8gpiCbBdVXeoagh4CLhywDIKFIj7vmc+0AxEgAXAKlXtUtUI8DzwoRTGaowxZoBUJoipQF3c+B5vWrw7ccmgHtgE3KyqMeB14AIRmSQiucBlwPREOxGR60RkjYisaWxsHOnfYIwxE1YqE4QkmKYDxpcCG4Aq4FTgThEpVNWtwH8AzwJPAxtxVxZHblD1x6q6WFUXl5eXj1DoxhhjUpkg9tD/rH8a7koh3rXAo+psB2qB+QCqeo+qnqaqF+Cqnt5KYazGGGMGSGWCWA3MFZEaEckErgIeH7DMbuAiABGpAOYBO7zxyV5/BvBh4MEUxmqMMWaAlL2hTlUjInITsALwAz9T1c0icoM3/27ga8DPRWQTrkrqNlVt8jbxGxGZBISBG1W1JVWxGmOMOVJKX2Gqqk8BTw2YdnfccD1wySDrnp/K2IwxxgzNnqQ2xhiTkCUIY4wxCVmCMMYYk5AlCGOMMQlZgjDGGJOQJQhjjDEJWYIwxhiTkCUIY4wxCVmCMMYYk5AlCGOMMQlZgjDGGJOQJQhjjDEJWYIwxhiTkCUIY4wxCVmCMMYYk5AlCGOMMQlZgjDGGJOQJQhjjDEJWYIwxhiTkCUIY4wxCVmCMMYYk5AlCGOMMQlZgjDGGJNQRroDMMaYsU5V6YnEiMSUmCqxmBJT3LAqqhCNKeFojJ5IjJ5wjJ5IlFDEG49EicYgPzuDguwMCrMDFOa4flaGDxFJKoZoTInEDvdjXl9RJhdkj/jvtgRhzDgVisRo6ughGI4SivYWWq6w6h0GyMzwuc7vIyvg9TN8BPw+QtEY3aEoXaEo3eEI3aEYXaEIwXCUnkgMnwg+AenX7x12cQh9A3293uV9Ivh8ccPe9Jgq3eFov/31jgcjUfKzMijODVCSm0lxToDi3ExK8tx4pt9HQ3sP+9uCHDgUZH9bsG/4QHuQYDgWF4eLL758Plyoe8cqEiPkHatUyPT7KMjOIMMvRGMQjcWIegkoEosRi3l9HXwb5QVZrP7H9414bJYgjBllqkpLV5japg52NHZS19JNhk/IzfSTl5Xh+pkZ5Ga5fmaGzyswtK/g6B2OxpSDnSH2tXaz71CQfYdcv741SFNHT7p/6ojyCeRmZpAd8NHRE+kr6JNRmpfJ5IIsphRlk5eZgeLO+lVBcSWvegVwZoaPrAw/WQGXKLMy/K4f8JHhk75E5vcdmRR7k23fOhk+sgJu2CdCR0+Ytu4IbcEwbcEI7cHD49Go4vcLfm/b/TpvWoZP8PulLw437iMv05+KQ57aBCEiy4DvAX7gp6r6zQHzi4D7gRleLN9S1Xu9eX8LfBZQYBNwraoGUxmvMccjFInRHgzT0ROhPej+6DuCbnjfoW52NHVS29TJjsZODnWH+9YTOVw4HY/8rAwqi7KpLM5hYWUhU4qyqSjMJidwuIDL9B8u+DIzfAjinTF71SFRd7YcisQIR2NkZvjIzfSTHfCTm5lBTsDfN56Z4YP4ahboV90C0PuzVPsXwr0Fc3RAVU005tYXLxnkBPzkZPrJCfgJ+KVfVUwwHKWlK0RLZ5jWrhAtXWFaukL0RGJ9yWBKYTblBVlkB1JTgI53KUsQIuIH7gIuBvYAq0XkcVXdErfYjcAWVb1CRMqBbSLyAFAOfBFYqKrdIvIwcBXw81TFa8aPaEw50BakrrmLPS3dXtfF/rYgPpG+s7reKpXes8SC7Iy+gqWi0HWF2Rn9CqWeSJTapk7eOtDB9gbXvdXQzu7mrmHPaCuLsqkpy+PykyuZVZ7PrLI8asrymFaSA0BXOEpXT5TOUORwPxShJxzDF3cW2Vsl4/eqZ0rzMqksyqYgO5DS4zrWZAf8VBblUFmUk+5Qxq2kEoSI/Ab4GfB7VU32um4JsF1Vd3jbeAi4EohPEAoUiPsLzAeagUhcbDkiEgZygfok92vGqFjMNfQFw1G6wlG6Q4frtLvCUYJeXXdP5MgGvp5wrK8ePRx1w+FojHDUNQyGIm7awY4Q9a3dRAZU2FYUZjGlKAe8xsbeeuX4+vhQ9Mj/2jkBPxWFWUwuyKaxo4ddBzv76oJ9AjNKc5kzuYB3n1BOcW4m+VmuEdL1AxR4jZJl+VnkZQ3951bo91E4wQp5M7YlewXxQ+Ba4P+JyCPAz1X1jWHWmQrUxY3vAc4csMydwOO4wr8A+BsvAe0VkW8Bu4Fu4BlVfSbRTkTkOuA6gBkzZiT5c8yxisWU5q4Qje09NHX0cLAjRFNHD01e/2BHD82dIbrDUYJhlwyC4SjB42jo8/ukrz434Pf1a1QN+H0E/ELA7yMvM4Np03O5/ORKppXkMq0kh+mluVQWZSdVxRAMRznQFuRAm2vgbGgLsv9QkAPtPTS0BZk/pYArTq5kTkUBcyfnU1OWZ1UXZlxLKkGo6nPAc16bwXLgWRGpA34C3K+q4QSrJbpva2BN61JgA/BeYLa33T/j2iyuBGqAVuAREblaVe9PENuPgR8DLF68eARqcie2nkiUXQe7qG3qZP+hYL+Gz/1el+hMO+AXyvKzmJSfSUluJpVFOWQHfGQHXH11VsBHdoYbzgn4XGNjpp/c3jpmr545p3f5jMN31GT4R+dxneyAn5mT8pg5KW9U9mfMWJd0G4SITAKuBq4B1gMPAOcBnwQuTLDKHmB63Pg0jqwmuhb4proWrO0iUgvMB2YCtara6O37UeAcXIO2SUI0pjR3hoD42w3pq0+PqVLX3OXq0Rs7eLuhk7cbO9jd3NXXwAiu4J9SlE1lYQ6LZhR7w9lMLszuSwhl+VlH1NUbY975km2DeBRXcN8HXKGq+7xZvxKRNYOsthqYKyI1wF5cI/PHBiyzG7gI+LOIVADzgB24suwsEcnFVTFdBAy2H5PAFx5cx1Ob9ie1bMAv1JTlsaCygMtPrmTO5HyqJ+VRVZzDpLxMfD4r+I2ZiJK9grhTVf+QaIaqLh5kekREbgJW4KqMfqaqm0XkBm/+3cDXgJ+LyCZcUrhNVZuAJhH5NbAO12i9Hq8ayQxve0MHT23azxWnVLGkuiTuVsPDtxuKCFXFOcyZnM/0kpxRq8YxxrxzJJsgFojIOlVtBRCREmC5qv5gqJVU9SngqQHT7o4brgcuGWTdrwJfTTK+caW5M8S2/e283djBuXPKqCk7ujrxn/55B1kZPu64YiGT8rNSFKUxZrxLNkF8TlXv6h1R1RYR+RwwZIIwQwuGo2yuP8S2/R28eaC9r2vqCPUtc9LUQh6/8bykq3ka2oM8um4vf7V4miUHY8xxSTZB+EREvMbk3ofgMlMX1vjV2RPhT9saeXrzfv74RgMdPe6xj9xMP3MrCnjPvMnMm1LACRUF7Gjs4I4ntvDkpn184JSqpLb/y5d3EY7F+Oz5s1L5M4wxE0CyCWIF8LCI3I27VfUG4OmURTXOHOoOs3LrAX7/+n5eeLORnkiMSXmZXHFKJRfOm8zCykKmFucccZVw7pwyHlpdx7dWbGPZiVPcqw2G0BWKcN+qXVyysOKoq6WMMWagZBPEbcD1wOdxjcnPAD9NVVDjQTSm/OGNBh74yy5efKuJSEyZUpjN8iUzWHbSFM6oLsU/TLWR3yd85dL5fOre1fzPX3bxqXNrhlz+4dV1HOoOc90Fs0fypxhjJqhkH5SL4Z6m/mFqw3nnO9QV5uE1dfxy1U7qmrupLMrmM+fXcOlJlZw8teiobxl99wnlnD1rEv/vD9v5yOnTBn3fTiQa456Xajl9ZgmnzywZiZ9ijJngkn0OYi7wDWAh0PdVClW1im7Pmwfa+fnLO3ls3V66w1GW1JTyD5cu4OKFFcd1C6mIu4q48q6X+Mmfa/m7i09IuNzTm/dT19zNP71/4THvyxhj4iVbxXQv7pbT7wDvwT0BbU9PAa+8fZA7//gWL20/SGaGjw+eWsUnz6nmxKqiEdvHKdOLef+7Kvnpn3dw9VkzjvhylKrykxd2UFOWx/sWVIzYfo0xE1uyp7Y5qroSEFXdpap34N6fNGGFIjH+/amtLP/JKt5u6OTWpfNYdftF/OdHTxnR5NDry0vn0ROJ8f2V24+Y95faZjbuOcRnz68Ztl3DGGOSlewVRFBEfMBb3tPRe4HJqQtrbNvR2MHND21g095DfPzMGfzT+xeSk6IvOvWqKctj+ZLpPPjqbj59Xk2/u5R+8sIOJuVl8pHTpqU0BmPMxJLsFcQtuG8yfBE4HffSvk+mKKYxS1V5ZE0dl3//RepauvjRNafz9Q+9K+XJodcXL5pLZoaPbz2zrW/aWwfaWflGA584u9pePW2MGVHDJgjvobi/VtUOVd2jqteq6kdUddUoxDdmtAXDfPGhDdz669c4eVoRv7/5fJaeOGVUY5hckM1nz5/F717bx8a6VgB++udasgM+rjl75qjGYowZ/4ZNEKoaBU6XCfwu57W7Wrjse3/mqU37uHXpPB747Flp+8zh586vYVJeJt/8/Rs0tAV5bP1e/ur06ZTm2YPtxpiRlWwbxHrgt97X5Dp7J6rqoymJagxZu6uZv/7RKqqKs3nkhrM5bUZ6nzEoyA7whffO4Y4ntnDT/6wnHIvxmfOGfoDOGGOORbIJohQ4SP87lxQY9wnimc0H8Ivw5E3nU5Q7Nr4X/LEzZ3LPS7W8urOZS0+aQrW9VsMYkwLJPkl9baoDGate3dnMydOKxkxyAMjM8HH7pQv4woPruf7d9loNY0xqJPsk9b0c+T1pVPXTIx7RGNIdirJpzyE+d8HYe2D8sndVct7cMgoHefWGMcYcr2SrmJ6MG84GPsSR35cedzbUtRKJKWdUj813G1lyMMakUrJVTL+JHxeRB4HnUhLRGLJ6ZzMicPrM0nSHYowxo+5Y3yI3F5gxkoGMRat3NjOvooCiHDtTN8ZMPMm2QbTTvw1iP+4bEeNWJBpj3a4WPmyvrzDGTFDJVjEVpDqQsWbrvnY6Q1HOqLHqJWPMxJRUFZOIfEhEiuLGi0XkgymLagx4dWczwJhtoDbGmFRLtg3iq6p6qHdEVVtx34cYt9bsbGZaSU7aXqlhjDHplmyCSLRcsrfIvuOoKqt3NrOk2qqXjDETV7IJYo2I/LeIzBaRWSLyHWBtKgNLp9qmTpo6Qiy2BGGMmcCSTRBfAELAr4CHgW7gxlQFlW5rdrYAsKTG2h+MMRNXUglCVTtV9Suqutjr/kFVO4dbT0SWicg2EdkuIl9JML9IRJ4QkY0isllErvWmzxORDXFdm4jcctS/7hi9urOZktwAs8vzR2uXxhgz5iR7F9OzIlIcN14iIiuGWccP3AVcCiwElovIwgGL3QhsUdVTgAuBb4tIpqpuU9VTVfVU3BfsuoDHkvtJx2/1zmYWV5cygT+BYYwxSVcxlXl3LgGgqi0M/03qJcB2Vd2hqiHgIeDKAcsoUOB9jCgfaAYiA5a5CHhbVXclGetxaWgLsutglzVQG2MmvGQTRExE+l6tISLVJHi76wBTgbq48T3etHh3AgtwL/7bBNysqrEBy1wFPDjYTkTkOhFZIyJrGhsbhwlpeKu99gd7QM4YM9ElmyD+EXhRRO4TkfuA54Hbh1knUf3MwKSyFNgAVAGnAneKSGHfBkQygQ8Ajwy2E1X9cW/bSHl5+XC/Y1irdzaTE/BzYlXh8AsbY8w4lmwj9dPAYmAb7k6mL+HuZBrKHmB63Pg0jnxF+LXAo+psB2qB+XHzLwXWqeqBZOIcCa/WNrNoRjEB/7G+x9AYY8aHZBupPwusxCWGLwH3AXcMs9pqYK6I1HhXAlcBjw9YZjeujQERqQDmATvi5i9niOqlkdYWDPPG/jbOsPYHY4xJuorpZuAMYJeqvgdYBAxZ4a+qEeAmYAWwFXhYVTeLyA0icoO32NeAc0RkEy4B3aaqTQAikgtczCh+93rdrhZiiiUIY4wh+ddlBFU1KCKISJaqviEi84ZbSVWfAp4aMO3uuOF64JJB1u0CJiUZ34hYvbMZv09YNKN4NHdrjDFjUrIJYo/3HMT/As+KSAvj8JOjq3e2cFJVIXlZ4/Y1U8YYk7RkvwfxIW/wDhH5I1AEPJ2yqNKgJxJlQ10r15w1M92hGGPMmHDUp8qq+nwqAkm3TXsOEYrErP3BGGM8di+np+8BOftAkDHGAJYg+qze2cys8jwm5WelOxRjjBkTLEEAsZiyxj4QZIwx/ViCAN5saKctGLH2B2OMiWP3cwKra5sBe0DOGHOcVCHcBV3N0N3s9VvccDgI+ZOhYAoUVLp+VkHy247FINQOwUP9u+5WQGHR1SP+cyxBAK/ubKGiMIvppTnpDsWY0dHTDm31ID4ongEZR9H2pgodB+DQHuhogM5Gr2uCzobDwz4/5JRCbumR/dxJUDQNiqZD1gh9mKu7FVp3Q9veuJgODoivESLd3mtDFTTmfk/vsPiheDpMmgOls2DSbG94tivQRVyh3FwLzTugpdYbroWWnW770Z7kY87Md9vNn+L+DSI9bv1I0A3H93vaXYyJ5E6yBJEKqsrq2mbOsA8ETWzBNoiGIW8EH95XdX/YoS6IRVzhggzo4wrprAJXoI6UaAQaNsO+ja4gb9vrEkJv19MWt7BA4VQorYGSaq9f4xJHZ5Mr+Pq6WmjZ5QrZgbIKIa8M8spd4RqLuDPo/ZtcP9iauIDLneT21dfNhJySw8cn/sXQIm4b7QdcMojveg4due3MfBdTbplLSFWnQCDPbUd89P079P6bxCLu9x3cDm89A9HQ4W0F8lwh3t3cfx95k90xqz7PXSEkSog5pW7dzkZo3wft+73+Aa+/zyWejCz3fyGv3A37s1w/IxuyCyG7GLKLDnc5ceMpMOETRCga470LJnP2rFF9q4dJt+Ah2L0Kdv4Zdr7oClKAE5bB4k/D7PcmV2Af2AKbHobaP0Oow1UvhLog3O2Gh/1sSpzsIlcwDuxyy6CwyhXiRVPd8MACobMJ6l6FPa9C3WqoX+ftH0Agv8KtN2kO1Lz78PY06p39emfBbz7tCrGBAnkueUyaA3Pe54aLZ7gCMa/cxRjIHvr3xWKuEO9qdvEequtfwB/YAtueTv4MPJAHJTNdHDPPPpxgCqdBvhdTZm5y20oYb9Ql1+a34aDXRbpd8iytcUmwpProqolyiqFs7rHHNMpE9Sj+A49xixcv1jVr1qQ7DJMK3S3Q9JY7E/YF3BlV75lVRubhcXBXAtGwO/vr7cfC7iqh7i8uIex/zZ2J+jNh6mJ39hcNwYYHXAFZPANO+yQsugYKKvrH0robNv3adQ2bXbXEjLPd1UcgFwI5Xj9u2J/hVWUQV6Xh9WNRV33Q3XJkF2x1BerARJNZ4Ar5gikunpZaN92XAVPeBdOWwPQlMPU0V43jDyR/rHva3dVCa507+y6pdklgNK6wYzFXTRX0rgb6lU9xw/kVA64yzLESkbWqujjhPEsQZlSoQqjTVW2Eh/iUSCzqCrymN/t3ic5qj4U/0xWe1ee6pDDtDFeI94qE4I0nYe29UPuCK3Dnvx8WfQJad8GmR2D3K27Z6WfCu/4KFn7QnbGmSjTsqiTa9rruUG910R5o2+eSxLQzXEKoWtT/9xgzDEsQJnW6W13B2brb1d227nIFWLDVnbH3HPL67a4642hkF0P5PCg74XBXPN3VE/c14IW8vteYJ+KuMPy9Xabr+wKuCmTywuQL0KbtLlFseMCdzQOUz3dJ4V0fdWfWxrzDWYIwh/XegXJgszuT16h3J4d3N0fMG4+FXV16qMOd+Yc6Idx5eLh9H7QkaBjMKnRVGjklrlEtq/DIfiCHIxoeD4+4qpPyea7xcixUIYSD8PYfXCPnlHeNjZiMGSFDJYgJ30g9rsWi7la8fRvdnST7N7m696OtrhG/uxUxM9/Vp2fmuvu4p591uJGweKYbzi4efwVoIBvmX5buKIwZdZYgxqtXfgB/+NrhO1l8AZi8AOYudWfBU05yZ/Pic3friK9/58uAzDzX+TPHX6FvjBmWJYjxKBqBF/7LJYTFn4HKk6FsnrvbxxhjkmQJYjza9aJ7mOfc78HCD6Q7GmPMO5QliHRRdbdMvv6oq+LxZ3pPTvbeeeN1J/81VJx4dNve8rhrK5jzvtTEboyZECxBpENXM/zu72DzY66BNzPfe6gr7sGuSMjdNbR3LXzqyeS3HYu6+/jnXnx8T5EaYyY8SxCjbcef4LHPu6dFL/q/cO4tg7/S4fn/gj/+m3sFQmlNctuv+4u7jXWBVS0ZY46PfQ9itER6YMU/wi+vdHcGffY5OP9LQ7/v59TlgMDGB5Pfz5bH3Qu+Tlh63CEbYyY2SxCj4cAW+Ml74ZU73V1F17/gXokwnKJpMOtC2PCge0fNcFRh6xMw56Kje4GYMcYkYAkilWIxWPVD+PGFrtrnYw/D5f99dG0Di66GQ7th5wvDL7t3nXs/j1UvGWNGQEoThIgsE5FtIrJdRL6SYH6RiDwhIhtFZLOIXBs3r1hEfi0ib4jIVhE5O5Wxjri2erj/Q/D0V9yroz//yrFV+8y/3L3aef0Dwy+75X/dA27zlh39fowxZoCUJQgR8QN3AZcCC4HlIrJwwGI3AltU9RTgQuDbItL7NNf3gKdVdT5wCrA1VbGOuM2PwQ/Odu/nv/w7sPzBY3/bZyAbTvoobH3c+7TgIFTdMrMudO9BMsaY45TKK4glwHZV3aGqIeAh4MoByyhQIO5TbvlAMxARkULgAuAeAFUNqWprCmMdGcFD8NgN8Min3KcKr/+z+/jM8b6mYtHH3dtKNz86+DL7N7l3+Fv1kjFmhKQyQUwF6uLG93jT4t0JLADqgU3AzaoaA2YBjcC9IrJeRH4qInmJdiIi14nIGhFZ09g4Qt8MOBa7XoYfngev/QrefRt8egWUzRmZbVedBuULhq5m2vJb9w6l+e8fmX0aYya8VCaIRKfNA98tvhTYAFQBpwJ3elcPGcBpwA9VdRHQCRzRhgGgqj9W1cWquri8PIUfbRlMJATP3QH3XuZuWf30M/Cefzi6L3gNR8Q1Vu9dAw1vJF5m6+PuAzh5ZSO3X2PMhJbKBLEHmB43Pg13pRDvWuBRdbYDtcB8b909qvoXb7lf4xLG2BINwwMfgRe/A6ddAze8CNPPSM2+Tv4b1wC94f4j5zW84b66ZtVLxpgRlMoEsRqYKyI1XsPzVcDjA5bZDVwEICIVwDxgh6ruB+pEZJ633EXAlhTGemx+f5v7LOUH7oQPfN99MyFV8svdq7o3/solpnhbfgsILLgidfs3xkw4KUsQqhoBbgJW4O5AelhVN4vIDSJyg7fY14BzRGQTsBK4TVWbvHlfAB4Qkddw1U//nqpYj8man8Gae+Dcm93Vw2hY9HH3io7tz/WfvvVxmHGW+zaxMcaMkJS+i0lVnwKeGjDt7rjheuCSQdbdACT8DF7a7XwJnroV5l4CF3119PY79xLIK4f198O8S920g2/Dgddh6TdGLw5jzIRgT1Ifrdbd8PA1UFIDH/np0O9SGmn+gGuLePNp6PQutLZ6tXZWvWSMGWGWII5GqBMeXO6+2Lb8IfeE82hbdDXEIu52WnDtD1NPh+LpQ69njDFHyRJEslThfz8PDVvgoz8buWccjtbkBe65iPUPuKuZ+vV295IxJiUsQSTrhf9yZ+sX/yvMTfOX2hZ9HBo2w8qvuXH7rKgxJgUsQSRj65Pwx6/DyVfB2TelOxo46SPumw+bHoYp74LSWemOyBgzDlmCGM7Bt+Gx6109/xXfO/73Ko2EnBJYcLkbXjDw9VbGGDMyLEEMZ+sTEOqAv/q5e7PqWLHkesgtg3d9JN2RGGPGKfsm9XDq10HxTCieke5I+ptxJvz92+mOwhgzjtkVxHD2roepY+81UMYYk2qWIIbS2eQ+91llCcIYM/FYghhK/XrXtysIY8wEZAliKHvXAQKVp6Q7EmOMGXWWIIZSvw7KToCsgnRHYowxo84SxGBU3RWEVS8ZYyYoSxCDadvrvr1gDdTGmAnKEsRgrIHaGDPBWYIYzN517hvQFSelOxJjjEkLSxCDqV8HFSeOrddrGGPMKLIEkYiqq2KqWpTuSIwxJm0sQSTSvAOCh6yB2hgzoVmCSGTvOte3BmpjzARmCSKR+vWQkQPlC9IdiTHGpI0liETq10HlyeC3t6EbYyYuSxADRSOwb6O1PxhjJjxLEAM1bYNwl93BZIyZ8CxBDGQN1MYYA6Q4QYjIMhHZJiLbReQrCeYXicgTIrJRRDaLyLVx83aKyCYR2SAia1IZZz/16yCrEEpnj9oujTFmLEpZK6yI+IG7gIuBPcBqEXlcVbfELXYjsEVVrxCRcmCbiDygqiFv/ntUtSlVMSa0dx1UnQo+u7gyxkxsqSwFlwDbVXWHV+A/BFw5YBkFCkREgHygGYikMKahRXrgwGZroDbGGFKbIKYCdXHje7xp8e4EFgD1wCbgZlWNefMUeEZE1orIdYPtRESuE5E1IrKmsbHx+CI+8DrEwtb+YIwxpDZBSIJpOmB8KbABqAJOBe4UkUJv3rmqehpwKXCjiFyQaCeq+mNVXayqi8vLy48v4t4GaruDyRhjUpog9gDT48an4a4U4l0LPKrOdqAWmA+gqvVevwF4DFdllVr16yG3DIqmD7+sMcaMc6lMEKuBuSJSIyKZwFXA4wOW2Q1cBCAiFcA8YIeI5IlIgTc9D7gEeD2FsTq9nxiVRBc/xhgzsaTsLiZVjYjITcAKwA/8TFU3i8gN3vy7ga8BPxeRTbgqqdtUtUlEZgGPubZrMoD/UdWnUxUrAD0d7iG5hQPb0Y0xZmJK6cuGVPUp4KkB0+6OG67HXR0MXG8HcEoqYzvC/tdAY9ZAbYwxHrvZv5c1UBtjTD+WIHrVr4PCaZA/Od2RGGPMmGAJotfedTDVrh6MMaaXJQiArmZoqbUnqI0xJo4lCHDPP4A1UBtjTBxLEHA4QVSemtYwjDFmLLEEAS5BTJoDOcXpjsQYY8YMSxDgveLbGqiNMSZeSh+Ue0eIhGD2e2DWe9IdiTHGjCmWIDIy4YM/SHcUxhgz5lgVkzHGmIQsQRhjjEnIEoQxxpiELEEYY4xJyBKEMcaYhCxBGGOMScgShDHGmIQsQRhjjElIVDXdMYwYEWkEdh3j6mVA0wiGM1IsrqNjcR0di+vojMe4ZqpqeaIZ4ypBHA8RWaOqi9Mdx0AW19GxuI6OxXV0JlpcVsVkjDEmIUsQxhhjErIEcdiP0x3AICyuo2NxHR2L6+hMqLisDcIYY0xCdgVhjDEmIUsQxhhjEprwCUJElonINhHZLiJfSXc8vURkp4hsEpENIrImzbH8TEQaROT1uGmlIvKsiLzl9UvGSFx3iMhe77htEJHLRjmm6SLyRxHZKiKbReRmb3paj9cQcaX7eGWLyKsistGL61+86ek+XoPFldbjFRefX0TWi8iT3nhKjteEboMQET/wJnAxsAdYDSxX1S1pDQyXIIDFqpr2h3JE5AKgA/ilqp7kTftPoFlVv+kl1hJVvW0MxHUH0KGq3xrNWOJiqgQqVXWdiBQAa4EPAp8ijcdriLj+mvQeLwHyVLVDRALAi8DNwIdJ7/EaLK5lpPF4xcX3d8BioFBVL0/V3+NEv4JYAmxX1R2qGgIeAq5Mc0xjjqq+ADQPmHwl8Atv+Be4wmZUDRJXWqnqPlVd5w23A1uBqaT5eA0RV1qp0+GNBrxOSf/xGiyutBORacD7gZ/GTU7J8ZroCWIqUBc3vocx8EfjUeAZEVkrItelO5gEKlR1H7jCB5ic5nji3SQir3lVUKNe9dVLRKqBRcBfGEPHa0BckObj5VWXbAAagGdVdUwcr0HigvT///ou8PdALG5aSo7XRE8QkmDamDhLAM5V1dOAS4EbveoUM7wfArOBU4F9wLfTEYSI5AO/AW5R1bZ0xJBIgrjSfrxUNaqqpwLTgCUictJox5DIIHGl9XiJyOVAg6quHY39TfQEsQeYHjc+DahPUyz9qGq9128AHsNVh40lB7x67d767YY0xwOAqh7w/rBjwE9Iw3Hz6qx/Azygqo96k9N+vBLFNRaOVy9VbQX+hKvnT/vxShTXGDhe5wIf8NooHwLeKyL3k6LjNdETxGpgrojUiEgmcBXweJpjQkTyvIZERCQPuAR4fei1Rt3jwCe94U8Cv01jLH16/0g8H2KUj5vXuHkPsFVV/ztuVlqP12BxjYHjVS4ixd5wDvA+4A3Sf7wSxpXu46Wqt6vqNFWtxpVXf1DVq0nV8VLVCd0Bl+HuZHob+Md0x+PFNAvY6HWb0x0X8CDucjqMu+r6DDAJWAm85fVLx0hc9wGbgNe8P5rKUY7pPFw15WvABq+7LN3Ha4i40n28TgbWe/t/Hfi/3vR0H6/B4krr8RoQ44XAk6k8XhP6NldjjDGDm+hVTMYYYwZhCcIYY0xCliCMMcYkZAnCGGNMQpYgjDHGJGQJwpgxQEQu7H0zpzFjhSUIY4wxCVmCMOYoiMjV3ncCNojIj7wXunWIyLdFZJ2IrBSRcm/ZU0Vklfdit8d6X+wmInNE5DnvWwPrRGS2t/l8Efm1iLwhIg94Tz8bkzaWIIxJkogsAP4G9yLFU4Eo8HEgD1in7uWKzwNf9Vb5JXCbqp6Me/q2d/oDwF2qegpwDu5pcHBvWL0FWIh7mv7cFP8kY4aUke4AjHkHuQg4HVjtndzn4F6KFgN+5S1zP/CoiBQBxar6vDf9F8Aj3ju2pqrqYwCqGgTwtveqqu7xxjcA1bgP1RiTFpYgjEmeAL9Q1dv7TRT55wHLDfX+mqGqjXrihqPY36dJM6tiMiZ5K4GPishk6PsO8Ezc39FHvWU+BryoqoeAFhE535t+DfC8um8w7BGRD3rbyBKR3NH8EcYky85QjEmSqm4RkX/CfenPh3uL7I1AJ3CiiKwFDuHaKcC9dvluLwHsAK71pl8D/EhE/tXbxl+N4s8wJmn2NldjjpOIdKhqfrrjMGakWRWTMcaYhOwKwhhjTEJ2BWGMMSYhSxDGGGMSsgRhjDEmIUsQxhhjErIEYYwxJqH/D1I/wU7SB0EFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.plot(history['accuracy'])\n",
    "plt.plot(history['precision'])\n",
    "plt.plot(history['recall'])\n",
    "plt.title('Train set_ model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'epoch')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAux0lEQVR4nO3deZxdZZ3v+8+3dk2pKVNV5jmEYA6GCBERlUG0G2jbaAsKjWlsQeS0OPTx3gb1tNrt6XvVo9fjPQ3GiHhptaFVoOXYICKK6EWGMCdkICRkIkORqapSSY2/88daVdkpdiW1k9rZldT3/Xrt117rWcP+rZ1U/ep5nrWeRxGBmZnZQJUUOwAzMzuxOHGYmVlenDjMzCwvThxmZpYXJw4zM8uLE4eZmeXFicPMzPLixGEnFUn3S7q62HEUm6QLJG0e4L5flvSjYz2PDR+lxQ7ATFJL1moV0AZ0pesfj4gfD/RcEXHJYMZ2JJJmAOuBsojoPJ6fbVYsThxWdBFR07Ms6RXg2oj4dd/9JJX6l7NZ8bmpyoasnmYSSTdK2gb8QNJoSb+Q1Chpd7o8JeuYhyVdmy5/RNIfJH0j3Xe9pH5rJOnnbJHULGm1pIvS8hJJN0l6WdJOST+RNCY97JH0fY+kFklvzeP6vizpp5J+lH7mC5JOlfQ5STskbZL0J1n7T5J0r6RdktZK+ljWthGS/r/0Ol8E3tznsyZJuiv93tZL+tRA4+xznjek3/EeSSskvTdr26WSXkyvZYuk/yMtr0//nfaksf9ekn/3nMD8j2dD3QRgDDAduI7k/+wP0vVpwH7gnw9z/FuA1UA98HXg+5LUdydJc4EbgDdHRC3wp8Ar6eZPAe8DzgcmAbuBm9Nt56XvoyKiJiL+mOf1/TnwQ2A08AzwQHqNk4F/BL6bte8dwOY0hsuA/6snuQFfAmanrz8Fevt50l/S/wt4Lj3vRcBnJP1pPoFKKkvP8ytgHPBJ4MfpdwfwfZKmxVrgdOA3afln07gbgPHA5wEPkncCc+Kwoa4b+FJEtEXE/ojYGRF3RURrRDQD/0TyC70/GyLiexHRBdwOTCT55dVXF1ABzJNUFhGvRMTL6baPA1+IiM0R0QZ8GbhM0mA09f4+Ih5Im+B+SvLL9asR0QHcCcyQNErSVODtwI0RcSAingVuBRan5/kg8E8RsSsiNgH/b9ZnvBloiIh/jIj2iFgHfA+4Is9YzwFq0vjaI+I3wC+AK9PtHSTfX11E7I6Ip7PKJwLTI6IjIn4fHl31hObEYUNdY0Qc6FmRVCXpu5I2SGoiaSoaJSnTz/HbehYiojVdrOm7U0SsBT5DkhR2SLpT0qR083TgnrSpZQ+wkiTR5EpA+dqetbwfeC1Ncj3rPfFOAnalybLHBpIaBOn2TX229ZgOTOqJP72Gzx9F/JOATRHR3U8MHwAuBTZI+l1Ws91/B9YCv5K0TtJNeX6uDTFOHDbU9f3L9LPAXOAtEVHHwaai1zU/5f1BEf8aEW8n+UUbwNfSTZuASyJiVNarMiK25IivUF4FxkiqzSqbBmxJl7cCU/ts67EJWN8n/tqIuPQoYpjap3+iN4aIeDIiFpE0Y/078JO0vDkiPhsRs0ia5v5LVhObnYCcOOxEU0vyl/ietIP6S4NxUklzJb1TUgVwIP2Mnr/8lwD/JGl6um+DpEXptkaS5rRZgxFHf9Lmp0eB/1tSpaT5wDVAz63KPwE+l948MIWk/6HHE0BT2vk/QlJG0umSDulAH4DHgX3A30kqk3QBSSK4U1K5pKskjUyb2ZpIvz9J75F0Stq31FPelfMT7ITgxGEnmv8BjABeAx4DfjlI560AvpqedxvJX82fT7d9G7iXpKmlOf3ct0Bv89c/Af9/2gx0ziDFk8uVwAySv/zvIen7eTDd9g8kzUbrSTqvf9hzUNr09efAgnT7ayT9IyPz+fCIaAfeC1ySnuMW4K8iYlW6y2LglbQJ8Xrgw2n5HODXQAvwR+CWiHg4n8+2oUXuozIzs3y4xmFmZnlx4jAbRJKWpA8C9n0tKXZsZoPFTVVmZpaXYTFWVX19fcyYMaPYYZiZnVCeeuqp1yKioW/5sEgcM2bMYNmyZcUOw8zshCJpQ65y93GYmVlenDjMzCwvThxmZpYXJw4zM8uLE4eZmeXFicPMzPLixGFmZnkpaOKQdHE6d/PaXJO3SFok6XlJz0paJuntRzpW0hhJD0p6KX0fXaj4H1q5nVseXluo05uZnZAKljjSGdluJhmCeR5wpaR5fXZ7CDgjIhYAHyUZ6vlIx94EPBQRc9LjCzab2CNrGlny8MtH3tHMbBgpZI3jbGBtRKxLx/G/E1iUvUNEtGTNPVzNwdnUDnfsIpK5o0nf31eoC6itLKOlrROP52VmdlAhE8dkDp0DeTMH5ybuJen9klYB/0FS6zjSseMjYitA+j4u14dLui5t/lrW2Nh4VBdQW1lKd8C+dk9WZmbWo5CJI9cc0K/70z0i7omI00hqDl/J59jDiYilEbEwIhY2NLxujK4Bqa0sA6D5QMdRHW9mdjIqZOLYDEzNWp9CMuVlThHxCDBbUv0Rjt0uaSJA+r5jMIPOVluZjAHZfKCzUB9hZnbCKWTieBKYI2mmpHLgCpJ5m3tlTWCPpDOBcmDnEY69F7g6Xb4a+HmhLqCmN3G4xmFm1qNgw6pHRKekG4AHgAxwW0SskHR9un0J8AHgryR1APuBD6Wd5TmPTU/9VeAnkq4BNgKXF+oa6lzjMDN7nYLOxxER9wH39SlbkrX8NeBrAz02Ld8JXDS4keZ2sI/DicPMrIefHD8M93GYmb2eE8dh+K4qM7PXc+I4jOryDJJrHGZm2Zw4DkMSNRWltLQ5cZiZ9XDiOIK6yjKa3FRlZtbLieMIaitL3VRlZpbFieMIksThGoeZWQ8njiOorSxzjcPMLIsTxxG4c9zM7FBOHEfgPg4zs0M5cRxB0lTV4cmczMxSThxHUFtZSkdX0NbZXexQzMyGBCeOI+gZIdfPcpiZJZw4jsAj5JqZHcqJ4whqKpIaR4sTh5kZ4MRxRB5a3czsUE4cR+Ch1c3MDuXEcQSucZiZHaqgiUPSxZJWS1or6aYc26+S9Hz6elTSGVnbPi1puaQVkj6TVf5lSVskPZu+Li3kNdSlNQ7fVWVmlijYnOOSMsDNwLuBzcCTku6NiBezdlsPnB8RuyVdAiwF3iLpdOBjwNlAO/BLSf8RES+lx30rIr5RqNiz1aQ1Dg87YmaWKGSN42xgbUSsi4h24E5gUfYOEfFoROxOVx8DpqTLbwAei4jWiOgEfge8v4Cx9itTIqrKM26qMjNLFTJxTAY2Za1vTsv6cw1wf7q8HDhP0lhJVcClwNSsfW9Im7dukzR6MIPOxUOrm5kdVMjEoRxlOQd8knQhSeK4ESAiVgJfAx4Efgk8B/T8yf8dYDawANgKfLOfc14naZmkZY2NjUd/FXhodTOzbIVMHJs5tJYwBXi1706S5gO3AosiYmdPeUR8PyLOjIjzgF3AS2n59ojoiohu4HskTWKvExFLI2JhRCxsaGg4pgvxCLlmZgcVMnE8CcyRNFNSOXAFcG/2DpKmAXcDiyNiTZ9t47L2+QvgjnR9YtZu7ydp1iqonhFyzcysgHdVRUSnpBuAB4AMcFtErJB0fbp9CfBFYCxwiySAzohYmJ7iLkljgQ7gE1md6F+XtICk2esV4OOFuoYetRWlbN7dWuiPMTM7IRQscQBExH3AfX3KlmQtXwtc28+x7+infPFgxjgQbqoyMzvIT44PgO+qMjM7yIljAGoryzjQ0U1HlydzMjNz4hgAj1dlZnaQE8cA9IyQ6zk5zMycOAakZzInD3RoZubEMSB1bqoyM+vlxDEAnszJzOwgJ44BcOe4mdlBThwDcDBxuMZhZubEMQCezMnM7CAnjgGoKM1QXlripiozM5w4BqyuspQmJw4zMyeOgfLQ6mZmCSeOAfIIuWZmCSeOAaqtLHXnuJkZThwDVlPhodXNzMCJY8CSPg7XOMzMnDgGyH0cZmYJJ44Bqq0so6Wtk67uKHYoZmZFVdDEIeliSaslrZV0U47tV0l6Pn09KumMrG2flrRc0gpJn8kqHyPpQUkvpe+jC3kNPer89LiZGVDAxCEpA9wMXALMA66UNK/PbuuB8yNiPvAVYGl67OnAx4CzgTOA90iakx5zE/BQRMwBHkrXC65nTg4nDjMb7gpZ4zgbWBsR6yKiHbgTWJS9Q0Q8GhG709XHgCnp8huAxyKiNSI6gd8B70+3LQJuT5dvB95XuEs4yEOrm5klCpk4JgObstY3p2X9uQa4P11eDpwnaaykKuBSYGq6bXxEbAVI38flOpmk6yQtk7SssbHxGC4j4aHVzcwSpQU8t3KU5exZlnQhSeJ4O0BErJT0NeBBoAV4DsjrN3ZELCVt+lq4cOEx92h7aHUzs0QhaxybOVhLgKQZ6tW+O0maD9wKLIqInT3lEfH9iDgzIs4DdgEvpZu2S5qYHjsR2FGg+A9xsKnKNQ4zG94KmTieBOZImimpHLgCuDd7B0nTgLuBxRGxps+2cVn7/AVwR7rpXuDqdPlq4OcFu4IsnnfczCxRsKaqiOiUdAPwAJABbouIFZKuT7cvAb4IjAVukQTQGREL01PcJWks0AF8IqsT/avATyRdA2wELi/UNWSrceIwMwMK28dBRNwH3NenbEnW8rXAtf0c+45+yncCFw1imAMyoixDpkTu4zCzYc9Pjg+QJA87YmaGE0deksThGoeZDW9OHHmorSjzk+NmNuw5ceSh1vOOm5k5ceTDfRxmZk4ceUkmc3Ifh5kNb04ceXCNw8zMiSMvtZWltLR1EuHJnMxs+HLiyENtZRld3cH+jq5ih2JmVjROHHnomczJzVVmNpw5ceTBQ6ubmTlx5KUuHVrdz3KY2XDmxJEHzwJoZubEkZeeyZxanDjMbBhz4siD+zjMzJw48uLJnMzMnDjyUlNeiuQah5kNb04ceSgpETXlHiHXzIY3J448ebwqMxvuCpo4JF0sabWktZJuyrH9KknPp69HJZ2Rte1vJa2QtFzSHZIq0/IvS9oi6dn0dWkhr6Gv2soyWtrcVGVmw1fBEoekDHAzcAkwD7hS0rw+u60Hzo+I+cBXgKXpsZOBTwELI+J0IANckXXctyJiQfq6r1DXkEuNaxxmNswVssZxNrA2ItZFRDtwJ7Aoe4eIeDQidqerjwFTsjaXAiMklQJVwKsFjHXA3FRlZsNdIRPHZGBT1vrmtKw/1wD3A0TEFuAbwEZgK7A3In6Vte8NafPWbZJG5zqZpOskLZO0rLGx8Viu4xCezMnMhrtCJg7lKMs5kYWkC0kSx43p+miS2slMYBJQLenD6e7fAWYDC0iSyjdznTMilkbEwohY2NDQcAyXcSjXOMxsuCtk4tgMTM1an0KO5iZJ84FbgUURsTMtfhewPiIaI6IDuBs4FyAitkdEV0R0A98jaRI7bmorS2luc+Iws+GrkInjSWCOpJmSykk6t+/N3kHSNJKksDgi1mRt2gicI6lKkoCLgJXpMROz9ns/sLyA1/A6dZVltHd209bpyZzMbHgqLdSJI6JT0g3AAyR3Rd0WESskXZ9uXwJ8ERgL3JLkBzrT5qXHJf0MeBroBJ4hveMK+LqkBSTNXq8AHy/UNeSSPZlTRU3meH60mdmQULDEAZDeKntfn7IlWcvXAtf2c+yXgC/lKF88yGHmJXto9fqaimKGYmZWFH5yPE89Q6v7ziozG66cOPLkyZzMbLgbUOKQ9GlJdUp8X9LTkv6k0MENRU4cZjbcDbTG8dGIaAL+BGgA/hr4asGiGsLq3FRlZsPcQBNHz8N8lwI/iIjnyP2A30kv+64qM7PhaKCJ4ylJvyJJHA9IqgW6CxfW0OVZAM1suBvo7bjXkAzxsS4iWiWNIWmuGnbKMiWMKMu4qcrMhq2B1jjeCqyOiD3pmFH/FdhbuLCGttrKUlo87IiZDVMDTRzfAVrTiZb+DtgA/EvBohriPNChmQ1nA00cnRERJCPWfjsivg3UFi6soa2msowmN1WZ2TA10D6OZkmfAxYD70hn9ysrXFhDW51rHGY2jA20xvEhoI3keY5tJBMy/feCRTXEJU1VrnGY2fA0oMSRJosfAyMlvQc4EBHDt4+josw1DjMbtgY65MgHgSeAy4EPAo9LuqyQgQ1lvqvKzIazgfZxfAF4c0TsAJDUAPwa+FmhAhvKaivLaG3vorOrm9KMx4k0s+FloL/1SnqSRmpnHseedHqeHnetw8yGo4HWOH4p6QHgjnT9Q/SZoGk4yR4hd1RVeZGjMTM7vgaUOCLi/5T0AeBtJIMbLo2Iewoa2RBWlyYOP8thZsPRgKeOjYi7gLsKGMsJo2cWwBbfWWVmw9Bh+ykkNUtqyvFqltR0pJNLuljSaklrJd2UY/tVkp5PX4+mQ5r0bPtbSSskLZd0h6TKtHyMpAclvZS+jz6aCz8WnszJzIazwyaOiKiNiLocr9qIqDvcsenT5TcDlwDzgCslzeuz23rg/IiYD3wFWJoeOxn4FLAwIk4HMsAV6TE3AQ9FxBzgoXT9uOqdk6PNTVVmNvwU8s6os4G1EbEuItqBO0nGuuoVEY9GxO509TFgStbmUmCEpFKgCng1LV8E3J4u3w68rzDh96+2dxZA1zjMbPgpZOKYDGzKWt+clvXnGuB+gIjYAnwD2AhsBfZGxK/S/cZHxNZ0v63AuFwnk3SdpGWSljU2Nh7ThfTlpiozG84KmThyTS0bOXeULiRJHDem66NJahYzgUlAdToPyIBFxNKIWBgRCxsaGvIK/EgqyzKUZ0qcOMxsWCpk4tgMTM1an8LB5qZekuYDtwKLImJnWvwuYH1ENEZEB3A3cG66bbukiemxE4Edfc95PHigQzMbrgqZOJ4E5kiaKamcpHP73uwdJE0jSQqLI2JN1qaNwDmSqiQJuAhYmW67F7g6Xb4a+HkBr6FfnszJzIarAT/Hka+I6JR0A/AAyV1Rt0XECknXp9uXAF8ExgK3JPmBzrR56XFJPwOeBjqBZ0jvuAK+CvxE0jUkCebyQl3D4dS4xmFmw1TBEgdARNxHn6FJ0oTRs3wtcG0/x34J+FKO8p0kNZCi8tDqZjZcDduBCo+Vm6rMbLhy4jhKtZVlHh3XzIYlJ46jVFtZ6kEOzWxYcuI4SnXpLIDd3TkfTTEzO2k5cRylmspSImBf+9Brrvrpsk2seHVvscMws5OUE8dRGqrjVbV1dvG5u1/g1t+vL3YoZnaScuI4SrVDdPrYl3fso7M7WP/avmKHYmYnKSeOo3SwxjG0OshXb0+mSVnX2EKE+1/MbPA5cRyl2t7pY4dWjWPV1mYgiWt369BKamZ2cnDiOEq1FUNzaPWV25p7l9e/1lLESMzsZOXEcZSGbFPVtibOmp7Mpruu0f0cZjb4nDiO0lCczGnXvna2N7XxrjeMp7RE7iA3s4Jw4jhKVeUZyjJiy+79xQ6l16ptScf46ZPrmDa2yjUOMysIJ46jJIk/e+NEfvbUZhqb24odDgCr0/6NuRNqmVVf7RqHmRWEE8cx+PS7TqW9q5vvPPxysUMBkjuqxlaX01BTwcz6atbv3OchUcxs0DlxHIOZ9dV84MzJ/OjxDWzdW/wmq1XbmzltYi2SmFlfQ3tnN68OgbjM7OTixHGMPvnOOUQEN/92bVHj6OoO1mxrZu74OiBJaoCbq8xs0DlxHKOpY6r40Jun8m9PbmLTrtaixbFxVyv7O7o4bWItALManDjMrDCcOAbBDRfOQRL/8zcvFS2G1ekdVW+YkNQ4xtVWUFWe8Z1VZjboCpo4JF0sabWktZJuyrH9KknPp69HJZ2Rls+V9GzWq0nSZ9JtX5a0JWvbpYW8hoGYMLKSD79lOnc9vYV1jcV5Wnvl1mZKBHPG1wCk/Ry+s8rMBl/BEoekDHAzcAkwD7hS0rw+u60Hzo+I+cBXgKUAEbE6IhZExALgLKAVuCfruG/1bI+I+wp1Dfn4zxfMpjxTwrcfKk6tY9W2JmbUV1NZluktc+Iws0IoZI3jbGBtRKyLiHbgTmBR9g4R8WhE7E5XHwOm5DjPRcDLEbGhgLEes4baCq4+dwb3Pvcqa7Y3H/mAQbZ6W3NvM1WPWfXVbN7dSltn13GPx8xOXoVMHJOBTVnrm9Oy/lwD3J+j/Argjj5lN6TNW7dJGp3rZJKuk7RM0rLGxsZ84j5qHz9vFtXlpXzrwTXH5fN67GvrZMOuVuZOqD2kfGZDNd1BUTvtzezkU8jEoRxlOZ9Gk3QhSeK4sU95OfBe4KdZxd8BZgMLgK3AN3OdMyKWRsTCiFjY0NCQd/BHY3R1OR99+0zuX76N5VuO39Sta7Y3EwGn9U0c9Ul/hzvIzWwwFTJxbAamZq1PAV7tu5Ok+cCtwKKI2Nln8yXA0xGxvacgIrZHRFdEdAPfI2kSGzKueftMRo4oO661jp6hRk7r01TlZznMrBAKmTieBOZImpnWHK4A7s3eQdI04G5gcUTk+k17JX2aqSRNzFp9P7B8UKM+RiNHlHHdebN4aNUOnt64+8gHDIJV25qpLs8wZfSI18VSX1PuxGFmg6pgiSMiOoEbgAeAlcBPImKFpOslXZ/u9kVgLHBLemvtsp7jJVUB7yZJLNm+LukFSc8DFwJ/W6hrOFofOXcGY6rLj1utY+XWJuZOqKWk5PWtgzPrq1nnxGFmg6i0kCdPb5W9r0/Zkqzla4Fr+zm2lSSp9C1fPMhhDrrqilL+5oLZ/Lf/WMldT23mA2flullscEQEq7c3c8npE3Nun1lfzW9XH5+bA8xsePCT4wXykXNn8NZZY/ncPS/w/OY9Bfuc7U1t7Gnt4A0Ta3Nun1lfQ2Nz25CbqdDMTlxOHAVSminhn//yTTTUVPDxHz5VsDk7VqZDjfTtGO/R00H+ymu+JdfMBocTRwGNrangu4vPYndrO5/416fp6Ooe9M/onbxpfO4aR89gh+teK85QKGZ28nHiKLDTJ4/kax+YzxPrd/HffvHioJ9/1dYmJo2sZGRVWc7t08ZUIfmWXDMbPAXtHLfEogWTeWHzXm79w3pOnzySyxdOPfJBA7RqWzOnTczdTAVQWZZh8qgRThxmNmhc4zhObrrkNN52yli+8O/LeXbTnkE5Z3tnNy83trxuqJG+ZtZX++lxMxs0ThzHSWmmhH++8kzG1VZw/Q+fYkfzgWM+57rXWujoitcNNdLXrHSU3AjPP25mx86J4zgaXV3O0sUL2bO/nU/8+GnaO4+ts7y/oUb6mllfTUtbJ40thbmzy8yGFyeO42zepDq+ftkZPPnKbm666/ljSh4rtzZTllHvnVP9mdWQDHa43s1VZjYInDiK4L1nTOKz7z6Vu5/ZwpXfe4wdTUfXbLVqWxOnjKulLHP4f0YPdmhmg8mJo0g+edEcbrnqTFZubeI9//MPPLUh/wERV29rPmL/BsCkUSMoLy1x4jCzQeHEUUSXvnEi9/zN2xhRnuGKpX/kjic2DvjYPa3tbN17YECJI1MiZoyt8mCHZjYonDiKbO6EWu79xNt56+x6Pnf3C3z+nhcG1O+xqqdj/DDPcGTz/ONmNlicOIaAkVVl/OAjb+ZvLpjNvz6+cUD9HgfvqDpyjQOSwQ437NxHV7dvyTWzY+PEMURkSsTfXXzaIf0ev1vT/3Doq7Y1MbqqjHG1FQM6/6z6ajq6gi279w9WyGY2TDlxDDE9/R61laVcfdsTfOqOZ3KOrLtyazNzJ9Qi5Zra/fVmerBDMxskThxD0NwJtdz36Xfwt+86lV8u38ZF33yYO57YSHfazNTdHazZ3nzEB/+y+ZZcMxssThxDVEVphk+/aw73f+YdzJtUx+fufoEPfvePrNnezKbdrbS2d/U7eVMuY6vLqa0sdeIws2NW0MQh6WJJqyWtlXRTju1XSXo+fT0q6Yy0fG46B3nPq0nSZ9JtYyQ9KOml9H10Ia+h2GY31HDHx87hG5efwcuNLVz67d/z9z9fAcDcPGocknrHrDIzOxYFSxySMsDNwCXAPOBKSfP67LYeOD8i5gNfAZYCRMTqiFgQEQuAs4BW4J70mJuAhyJiDvBQun5Sk8RlZ03hoc9ewKIFk3lkTSMSnDq+Jq/zeJRcMxsMhZyP42xgbUSsA5B0J7AI6J3NKCIezdr/MWBKjvNcBLwcERvS9UXABeny7cDDwI2DGfhQNaa6nG9+8AwuXziF7U0HqCrP759vZn0NP3/uVQ50dFFZlilQlGZ2sitk4pgMbMpa3wy85TD7XwPcn6P8CuCOrPXxEbEVICK2Shp3rIGeaM6ZNfaojpvVUE0EbNjZesQ5PMzM+lPIPo5c94nmfPpM0oUkiePGPuXlwHuBn+b94dJ1kpZJWtbY2P/zEMPJwTurfEuumR29QiaOzUD2HKlTgFf77iRpPnArsCgidvbZfAnwdERszyrbLmlieuxEYEeuD4+IpRGxMCIWNjQ0HMNlnDx6EofHrDKzY1HIxPEkMEfSzLTmcAVwb/YOkqYBdwOLI2JNjnNcyaHNVKTnuDpdvhr4+aBGfRKrrihlfF2F5+Uws2NSsD6OiOiUdAPwAJABbouIFZKuT7cvAb4IjAVuSZ+A7oyIhQCSqoB3Ax/vc+qvAj+RdA2wEbi8UNdwMvJgh2Z2rArZOU5E3Afc16dsSdbytcC1/RzbSpJU+pbvJLnTyo7CzPoafrVi2+vKI4LdrR1sbzrA2JpyxtVWFiE6MzsRFDRx2NAzq76anfva+Yf/tYLG5ja2Nx1gW9MBtje19Q7nXpYRl501lb+5YDZTx1QVOWIzG2qcOIaZN00bhQR3PLGRCXWVjK+r5E1TRzNhZLI8vq6Cx9ft4t+e3MRPl23iL86czCcuPIXpYw8/r7mZDR+KOPnnZ1i4cGEsW7as2GEMGW2dXZRnSg47su62vQdY8ruXueOJjXR2B4sWTOKGC09hVkN+T6ub2YlL0lM9/c6HlDtx2OHsaDrA0kfW8aPHN9De2c2fnzGJDy2cyptnjqEs4zEyzU5mThxOHMfktZY2vvf7dfzwjxtobe+irrKUd542jnfPm8B5p9ZTW1lW7BDNbJA5cThxDIrW9k4eWfMav165nd+s2sGufe2UZ0o4Z/ZY3j1vPBedNo5Jo0YUO0wzGwROHE4cg66rO3hqw24efHEbD764nVd2tgLJsyLnzBrLObPG8NbZY31rr9kJyonDiaOgIoKXG1t4eHUjf3x5J0+s30VzWycAsxuqeevssbx1Vj3Tx1ZRXlpCeaYkee95ZUqoKD18h72ZHV9OHE4cx1VnVzcrXm3ij+t28seXd/LkK7tobe867DGlJeI/TarjrOljWDhjNAunj2ZcnWsrZsXixOHEUVQdXd0s37KXxuY22ru6ae9MX+lyW2c3Tfs7eGbTHp7btIe29GHEqWNGsHD6GM6cPprTJ9UxadQI6msqyJS4ZmJWaP0lDj8AaMdFWaaEN00b2Cy/7Z3dvLi1iWWv7OKpDbv5w9rXuOeZLb3bMyVifG0FE0ZWMnHkCCaMrGRCXSUjyjOUlojSTAllGVFaUkKmRJRlRE1FKfMm1eV999fe/R2s3dHCiLIMo6vLGDWinBHlngTLhjfXOGzIiwg27drPSzua2dZ0gG17D7B1b8/7frbuPXDEZjAAKRly5Ywpo5g/ZSTzp45i3sS63tkQW9o6WbFlLy9s2cvzm5P3XANCVpSWMKoqSSKjqsqYNGoEC6aO4k3TRnHahDrKS/18i50c3FTlxHHSigha2jrZ39FFZ1fQ2RV0dHcn713ddHUHu1rbeWHzXp7fvIfnNidNZpD0q8ydUEtbZzcvN7bQ8+MwedQI3jh5JG+cMpLTJtTS3tnNnv0d7G5tZ29rB3tak+U9+zt45bV97EjPV1Fawhsnj+RN00bxpmmjWTB1FBNHVrrT305IThxOHJaKCLY1HeC5TUkieWHLXsozJcxPayKnTx5JQ21FXufbuvcAz2zcwzMbd/PMpuScPYNG1tdU9J73jelrfF2Fk4kNeU4cThx2HLV3drNyaxPPbNzN81v2snzLXtbuaKE7/XHrSSZnTBnFeafWc8aUUZS4w9+GGCcOJw4rstb2Tl58tYkX0n6U5Vv28tKOpHlsbHU5589t4MK54zjv1AZGjvAQLlZ8vqvKrMiqyktZOGMMC2eM6S3bva+dR15q5DerdvCbVTu4++ktZErEWdNH887TxjGzvrr3duXkFuau3uXO7mBsTTkT6g7eXTa2utw1Fys41zjMhoiu7uDZTbvTJNLIyq1NeZ+jLCPG11UycWQlU0ZXcdqEWk6bWMcbJtTSUOt+FcuPm6qcOOwEs73pAI3NbVSUllBRmqG8NBmWpee9RGLnvvbe25K3NR16m/KGna1s3Xug93xjqsuTRDKhjtMmJolkRFkmeZUn75XpcmVp8gyME83wVpSmKkkXA98GMsCtEfHVPtuvAm5MV1uA/xwRz6XbRgG3AqcDAXw0Iv4o6cvAx4DG9LjPp3Obm51UxqczNB5OQ20FDbUVvHHKyJzb97S2s2pbM6u2NrFqWzMrtzXzr09s4EBH94Bi6HmQMnmwMn24skSMrang1PG1nDahlrkTknfXaIaPgiUOSRngZuDdwGbgSUn3RsSLWbutB86PiN2SLgGWAm9Jt30b+GVEXCapHMie/PpbEfGNQsVudrIYVVWejlQ8tresqzvYuKuVPa3t7O/o4kBHF/vbu9nf0ZWstyfvnd1BZ1fSl9LRlTwX09ndTUdXsL3pAI+81MhdT2/uPe/oqjLmTqjl1PG11FWWHTKAZXlWTaksk9RmSkuUvpdQUsIhT/qXliRP/5dlSijNiPJMCaWZEirLSqgqz+/XVltnF+tf28f6xn2Ul5YwurqcsdXljK4up7aidEDJrrOr2zWwLIWscZwNrI2IdQCS7gQWAb2JIyIezdr/MWBKum8dcB7wkXS/dqC9gLGaDRuZEjGzvho49nnkd7a0sXp7M6u3Ja9V25q555kttLZ30dVdmGbwUVVlTB1dxdQxI5g6uoopY6qYMjpZjgjWbG9hzfbm3tcrO1v7jaW0RL2JpKo8Q1tnNwc6ujjQ0U1bZxf727s40Jk8RDp51AguSO98O/eUsXknsP60dXb1Js0TRSETx2RgU9b6Zg7WJnK5Brg/XZ5F0hT1A0lnAE8Bn46InvEfbpD0V8Ay4LMRsbvvySRdB1wHMG3atGO5DjPrx9iaCs6tqeDc2fWv29bZ1X3IgJZt6aCWPbWX7gg6u4Ou7uRp/67upEbTldZwOroO1nTau7rp7OqmtaOLLbv3s2n3flZta+bXK3f0PmiZTYLpY6o4dXwtl5w+kTnja5jdUENnd7B7Xzu79rWzu/XQ931tXYyuKqGyLENFWfJeWZqhsiypLa3c2sS/P7OFHz++kfJMCW+ZNYYL5o7jwrkNzKyvzlkbifQad+9rZ+OuVjbuamXDzlY29Szvau0dxaC2opS6EWXUVibvdZVl1I0oTd/LqKssZeSIZHlk1vaIZEy1pgMdNO3vTN/T14FOPnzOdE4ZVzOo/+4F6xyXdDnwpxFxbbq+GDg7Ij6ZY98LgVuAt0fETkkLSWogb4uIxyV9G2iKiL+XNB54jaTf4yvAxIj46OFicee42cmpuztobGlj065WNu1OJhKbM66WU8bV9I5BNpjaO7tZ9soufrt6B79d3cjaHS0ATKirpLy0hPbOJDG2d3bTlibJvr9iJZhYV8m0sVVMG1PFlNFVdEcc+ks/Kwns3d9B84HOo4q3trKUW646k3fMaTiq44vROb4ZmJq1PgV4NUdg80k6wS+JiJ1Zx26OiMfT9Z8BNwFExPasY78H/GLwQzezE0FJiXpvIsh+PqZQyktLOPeUes49pZ4v/Bls2tXKw2saeeqVXb3by9N+nPLSEirS97oRZUwdU8X0MVVMHj2CitL8klpXdzIeW9P+JJH0JJe9+zsokXLWUGoqSgvW/FXIxPEkMEfSTGALcAXwl9k7SJoG3A0sjog1PeURsU3SJklzI2I1cBFp34ikiRGxNd31/cDyAl6DmVm/po6pYvE501l8zvSCfk6mRIxMm6imHnn3gitY4oiITkk3AA+Q3I57W0SskHR9un0J8EVgLHBL2j7YmVUt+iTw4/SOqnXAX6flX5e0gKSp6hXg44W6BjMzez0/AGhmZjn118fhGWfMzCwvThxmZpYXJw4zM8uLE4eZmeXFicPMzPLixGFmZnkZFrfjSmoENhzl4fUkQ5wMNY4rP44rP44rP0M1Lji22KZHxOvGKxkWieNYSFqW6z7mYnNc+XFc+XFc+RmqcUFhYnNTlZmZ5cWJw8zM8uLEcWRLix1APxxXfhxXfhxXfoZqXFCA2NzHYWZmeXGNw8zM8uLEYWZmeXHiOAxJF0taLWmtpJuKHU8PSa9IekHSs5KKNl68pNsk7ZC0PKtsjKQHJb2Uvo8eInF9WdKW9Dt7VtKlRYhrqqTfSlopaYWkT6flRf3ODhNXUb8zSZWSnpD0XBrXP6Tlxf6++our6P/H0jgykp6R9It0fdC/L/dx9ENSBlgDvJtkKtsngSsj4sWiBkaSOICFEVHUB44knQe0AP8SEaenZV8HdkXEV9NkOzoibhwCcX0ZaImIbxzPWPrENRGYGBFPS6oFngLeB3yEIn5nh4nrgxTxO1Myu1t1RLRIKgP+AHwa+AuK+331F9fFFPn/WBrffwEWAnUR8Z5C/Ey6xtG/s4G1EbEuItqBO4FFRY5pSImIR4BdfYoXAbeny7eT/AI6rvqJq+giYmtEPJ0uNwMrgckU+Ts7TFxFFYmWdLUsfQXF/776i6voJE0B/gy4Nat40L8vJ47+TQY2Za1vZgj8MKUC+JWkpyRdV+xg+hjfMyd8+j6uyPFku0HS82lT1nFvQssmaQbwJuBxhtB31icuKPJ3lja7PAvsAB6MiCHxffUTFxT//9j/AP4O6M4qG/Tvy4mjf8pRNiT+qgDeFhFnApcAn0ibZuzwvgPMBhYAW4FvFisQSTXAXcBnIqKpWHH0lSOuon9nEdEVEQuAKcDZkk4/3jHk0k9cRf2+JL0H2BERTxX6s5w4+rcZmJq1PgV4tUixHCIiXk3fdwD3kDSrDRXb0zbznrbzHUWOB4CI2J7+sHcD36NI31naJn4X8OOIuDstLvp3liuuofKdpbHsAR4m6Uco+veVK64h8H29DXhv2gd6J/BOST+iAN+XE0f/ngTmSJopqRy4Ari3yDEhqTrtwERSNfAnwPLDH3Vc3QtcnS5fDfy8iLH06vnBSb2fInxnaafq94GVEfH/ZG0q6nfWX1zF/s4kNUgalS6PAN4FrKL431fOuIr9fUXE5yJiSkTMIPl99ZuI+DCF+L4iwq9+XsClJHdWvQx8odjxpDHNAp5LXyuKGRdwB0mVvIOkhnYNMBZ4CHgpfR8zROL6IfAC8Hz6gzSxCHG9naS583ng2fR1abG/s8PEVdTvDJgPPJN+/nLgi2l5sb+v/uIq+v+xrBgvAH5RqO/Lt+OamVle3FRlZmZ5ceIwM7O8OHGYmVlenDjMzCwvThxmZpYXJw6zIU7SBT0jnZoNBU4cZmaWFycOs0Ei6cPpPA3PSvpuOhBei6RvSnpa0kOSGtJ9F0h6LB0Q756eAfEknSLp1+lcD09Lmp2evkbSzyStkvTj9Glvs6Jw4jAbBJLeAHyIZADKBUAXcBVQDTwdyaCUvwO+lB7yL8CNETGf5GnjnvIfAzdHxBnAuSRPwEMyYu1ngHkkowe8rcCXZNav0mIHYHaSuAg4C3gyrQyMIBlMrhv4t3SfHwF3SxoJjIqI36XltwM/TccgmxwR9wBExAGA9HxPRMTmdP1ZYAbJBEJmx50Th9ngEHB7RHzukELp7/vsd7gxfg7X/NSWtdyFf3atiNxUZTY4HgIukzQOeud5nk7yM3ZZus9fAn+IiL3AbknvSMsXA7+LZA6MzZLel56jQlLV8bwIs4HwXy1mgyAiXpT0X0lmZiwhGZn3E8A+4D9JegrYS9IPAsnw1kvSxLAO+Ou0fDHwXUn/mJ7j8uN4GWYD4tFxzQpIUktE1BQ7DrPB5KYqMzPLi2scZmaWF9c4zMwsL04cZmaWFycOMzPLixOHmZnlxYnDzMzy8r8BM4tl1K0s/94AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history['loss'])\n",
    "plt.title('Train set_ model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We forgot to save the validation metrics.  \n",
    "- These can be viewed in the model training progress.  \n",
    "- Regardless:\n",
    "    - We can see that the greatest loss minimiation occurred before the 4th epoch and the accuracy, precision and recall likewise have the greatest maximization before the 4 th epoch.    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}