FS-Permutation+Wrapper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f2178a5a",
   "metadata": {},
   "source": [
    "# A Two Stage Strategy for Feature Subset Selection\n",
    "1. Use Permutation Feature Importance to identify a large subset of candidates\n",
    "2. Use a Wrapper search on this subset to select a final set"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec35a629",
   "metadata": {},
   "source": [
    "## Permutation Feature Importance\n",
    "https://scikit-learn.org/stable/modules/permutation_importance.html  \n",
    "\n",
    "This is a good way to score feature importance in the context of a specific classifier/model.  \n",
    "The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled.  \n",
    "The idea comes from the original work on Random Forests by Leo Breiman:  \n",
    "L. Breiman, “Random Forests”, Machine Learning, 45(1), 5-32, 2001."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "39ad8bc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.inspection import permutation_importance\n",
    "from sklearn import preprocessing\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71fcf18b",
   "metadata": {},
   "source": [
    "Load the Ionsphere dataset from the UCI repository. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c6578ead",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(351, 35)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.99539</td>\n",
       "      <td>-0.05889</td>\n",
       "      <td>0.85243</td>\n",
       "      <td>0.02306</td>\n",
       "      <td>0.83398</td>\n",
       "      <td>-0.37708</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.03760</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.51171</td>\n",
       "      <td>0.41078</td>\n",
       "      <td>-0.46168</td>\n",
       "      <td>0.21266</td>\n",
       "      <td>-0.34090</td>\n",
       "      <td>0.42267</td>\n",
       "      <td>-0.54487</td>\n",
       "      <td>0.18641</td>\n",
       "      <td>-0.45300</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>-0.18829</td>\n",
       "      <td>0.93035</td>\n",
       "      <td>-0.36156</td>\n",
       "      <td>-0.10868</td>\n",
       "      <td>-0.93597</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>-0.04549</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.26569</td>\n",
       "      <td>-0.20468</td>\n",
       "      <td>-0.18401</td>\n",
       "      <td>-0.19040</td>\n",
       "      <td>-0.11593</td>\n",
       "      <td>-0.16626</td>\n",
       "      <td>-0.06288</td>\n",
       "      <td>-0.13738</td>\n",
       "      <td>-0.02447</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>-0.03365</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.00485</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>-0.12062</td>\n",
       "      <td>0.88965</td>\n",
       "      <td>0.01198</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.40220</td>\n",
       "      <td>0.58984</td>\n",
       "      <td>-0.22145</td>\n",
       "      <td>0.43100</td>\n",
       "      <td>-0.17365</td>\n",
       "      <td>0.60436</td>\n",
       "      <td>-0.24180</td>\n",
       "      <td>0.56045</td>\n",
       "      <td>-0.38238</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>-0.45161</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.71216</td>\n",
       "      <td>-1.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.90695</td>\n",
       "      <td>0.51613</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>-0.20099</td>\n",
       "      <td>0.25682</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>-0.32382</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>-0.02401</td>\n",
       "      <td>0.94140</td>\n",
       "      <td>0.06531</td>\n",
       "      <td>0.92106</td>\n",
       "      <td>-0.23255</td>\n",
       "      <td>0.77152</td>\n",
       "      <td>-0.16399</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.65158</td>\n",
       "      <td>0.13290</td>\n",
       "      <td>-0.53206</td>\n",
       "      <td>0.02431</td>\n",
       "      <td>-0.62197</td>\n",
       "      <td>-0.05707</td>\n",
       "      <td>-0.59573</td>\n",
       "      <td>-0.04608</td>\n",
       "      <td>-0.65697</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1        2        3        4        5        6        7        8  \\\n",
       "0  1  0  0.99539 -0.05889  0.85243  0.02306  0.83398 -0.37708  1.00000   \n",
       "1  1  0  1.00000 -0.18829  0.93035 -0.36156 -0.10868 -0.93597  1.00000   \n",
       "2  1  0  1.00000 -0.03365  1.00000  0.00485  1.00000 -0.12062  0.88965   \n",
       "3  1  0  1.00000 -0.45161  1.00000  1.00000  0.71216 -1.00000  0.00000   \n",
       "4  1  0  1.00000 -0.02401  0.94140  0.06531  0.92106 -0.23255  0.77152   \n",
       "\n",
       "         9  ...       25       26       27       28       29       30  \\\n",
       "0  0.03760  ... -0.51171  0.41078 -0.46168  0.21266 -0.34090  0.42267   \n",
       "1 -0.04549  ... -0.26569 -0.20468 -0.18401 -0.19040 -0.11593 -0.16626   \n",
       "2  0.01198  ... -0.40220  0.58984 -0.22145  0.43100 -0.17365  0.60436   \n",
       "3  0.00000  ...  0.90695  0.51613  1.00000  1.00000 -0.20099  0.25682   \n",
       "4 -0.16399  ... -0.65158  0.13290 -0.53206  0.02431 -0.62197 -0.05707   \n",
       "\n",
       "        31       32       33  Class  \n",
       "0 -0.54487  0.18641 -0.45300      g  \n",
       "1 -0.06288 -0.13738 -0.02447      b  \n",
       "2 -0.24180  0.56045 -0.38238      g  \n",
       "3  1.00000 -0.32382  1.00000      b  \n",
       "4 -0.59573 -0.04608 -0.65697      g  \n",
       "\n",
       "[5 rows x 35 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_heads = [str(x) for x in range(34)]\n",
    "col_heads.append('Class')\n",
    "ion_data = pd.read_csv('ionosphere.csv', names = col_heads, na_values='?', header = None)\n",
    "ion_data.dropna(axis = 0, inplace = True)\n",
    "print(ion_data.shape)\n",
    "ion_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31f199b0",
   "metadata": {},
   "source": [
    "Split the data 50:50; 50% of the data will be held back from the feature selection and model training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e0757455",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = ion_data.pop('Class').values\n",
    "X = ion_data\n",
    "ion_scaler = preprocessing.MinMaxScaler().fit(X)\n",
    "X_scaled = ion_scaler.transform(X)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, \n",
    "                                                    random_state=2, test_size=1/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "94efc1d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Final results dataframe\n",
    "res_df = pd.DataFrame([['All Features',0.0,0.0],\n",
    "                   ['After PI',0.0,0.0],\n",
    "                   ['After Wrapper',0.0,0.0]],columns=['Method','X-Val','Hold-Out'])\n",
    "res_df.set_index('Method', inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baa7d3e9",
   "metadata": {},
   "source": [
    "Calculating the base-line accuracies on the full feature set.   \n",
    "Accuracy for the training data is assessed using cross-validation.  \n",
    "Accuracy on the test set is assessed using a simple hold-out strategy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cc09c142",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_Val on training all features: 0.818\n",
      "Hold Out testing all features: 0.795\n"
     ]
    }
   ],
   "source": [
    "kNN = KNeighborsClassifier(n_neighbors=3)\n",
    "kNN = kNN.fit(X_train,y_train)\n",
    "y_pred = kNN.predict(X_test)\n",
    "acc = accuracy_score(y_pred,y_test)\n",
    "cv_acc = cross_val_score(kNN, X_train, y_train, cv=8)\n",
    "\n",
    "res_df.loc['All Features']['X-Val']=cv_acc.mean()\n",
    "res_df.loc['All Features']['Hold-Out']=acc\n",
    "\n",
    "print(\"X_Val on training all features: {0:.3f}\".format(cv_acc.mean())) \n",
    "print(\"Hold Out testing all features: {0:.3f}\".format(acc)) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd93fa2e",
   "metadata": {},
   "source": [
    "## Permutation feature importance stage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2bc7dfd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "knn_perm = permutation_importance(kNN, X_train, y_train, \n",
    "                                n_repeats=10, random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e3184727",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAGpCAYAAACK4V1rAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAnGUlEQVR4nO3dfbxeZX3n+89XQEFAUAGlECY+IFOLEDTl0FJtRNtSpaC2tlJLcfQl1aOtUPtAtfXheJzjqU+1zkxbKhScQVoqIApYyLEGtBUswUBCg+Io0kiGYD0CoqKB3/yxVvRmZ++dnb2z1tphfd6v1/3a614P93WtBH575bqv3+9KVSFJGo9HDN0BSVK/DPySNDIGfkkaGQO/JI2MgV+SRmbXoTswF/vtt18tXbp06G507r777mPPPfccuhuDGOu9j/W+wXvv495Xr179jaraf+r+nSLwL126lOuvv37obnRu1apVrFixYuhuDGKs9z7W+wbvvY97T/K16fY71CNJI2Pgl6SRMfBL0sgY+CVpZAz8kjQyBn5JGhkDvySNjIFfkkbGwC9JI2Pgl6SRMfBL0sgY+CVpZAz8kjQyBn5JGhkDvySNjIFfkkYmVTV0H7Zp+fLlNYaFWD54/qW8d+1OsTbODvfGZ2we5b2P9b7Be5/rvd/2rhfOu50kq6tq+dT9gz3xJ9klyReSXDZUHyRpjIYc6nkDsH7A9iVplAYJ/EkOBl4IfGiI9iVpzIZ64v8z4A+AB2c6IclpSa5Pcv1dd93VW8ck6eGu98Cf5ARgU1Wtnu28qjqrqpZX1fL999+/p95J0sPfEE/8xwInJrkN+FvguCT/Y4B+SNIo9R74q+qPqurgqloKvAz4x6r6jb77IUljNc5JtIvUMw7ah9tevmLobgxi1apVo7z3sd43eO9D3vuggb+qVgGrhuyDJI2NT/yLyNqv380rzrx86G4M4o3P2PywuveFZFtKXRtiVs+SJJ9Osj7JzUne0HcfJGnMhnji3wy8sapuSLI3sDrJyqr61wH6IkmjM8Ssno1VdUO7fS9N2YaD+u6HJI3VoGWZkywFjgKum+aYmbuS1IEhq3PuBVwEnF5V90w9buauJHVjqCJtu9EE/fOr6uIh+iBJYzXErJ4AZwPrq+p9fbcvSWM3xKyeY4FTgLVJ1rT73lRVVwzQl0XFzN0VQ3dDGoXeA39VfRZI3+1Kkhpm7i4iY8jcNaNVGl5nY/zbytBN8ntJKsl+XfVBkrS1Lp/4Z8zQTbIE+Dng9g7blyRNo7Mn/m1k6L6fZunF6qp9SdL0epnOOZmhm+RE4OtVdeM2rjFzV5I60Hngn8zQpRn+eTPwlm1dZ+auJHWj08A/TYbuU4AnATe2a+4eDNyQ5Ild9kOS9COdfbk7XYZuVa0FDpg45zZgeVV9o6t+SJIeqstZPWbobqcxZ+5K6k9ngX8uGbpVtbSr9iVJ0zNzdxF5uGbumq0rLS5DlWXeN8lHk9zSZvb+1BD9kKQxGuqJ/wPAP1TVryR5JPDogfohSaPTe+BP8hjgOcArAKrq+8D3++6HJI3VEEM9TwbuAv4myReSfCjJnlNPMnNXkroxRODfFXgm8BdVdRRwH3Dm1JPM3JWkbgwR+DcAG6rquvb9R2l+EUiSetB74K+q/wX8W5LD2l3PA/61735I0lgNNavnt4Hz2xk9XwH+00D9WFTM3JXUh0ECf1WtAZYP0bYkjd0gCVySpOFYsmERsWSDpD4MVbLh+CRfTPLlJFtN5ZQkdaf3wJ9kF+C/Ar8IPB04OcnT++6HJI3VEE/8RwNfrqqvtOUa/hY4aYB+SNIoDRH4DwL+beL9hnbfQ1iyQZK6MUTgn25xltpqhyUbJKkTQ5VsWDLx/mDgjgH6IUmjNETg/xfg0CRPajN3XwZ8fIB+SNIo9T6Pv6o2J3k9cCWwC3BOVd3cdz8WI0s2SOrDUCUbrgCuGKJtSRo7M3cXkcWWuWvGrfTw1NkYf5JzkmxKsm6aY7+XpJLs11X7kqTpdfnl7rnA8VN3JlkC/Bxwe4dtS5Jm0Fngr6prgG9Oc+j9wB8wzdx9SVL3ep3OmeRE4OtVdeMczjVzV5I60FvgT/Jo4M3AW+Zyvpm7ktSNPp/4nwI8CbgxyW00Gbs3JHlij32QpNHrbTpnVa0FDtjyvg3+y6vqG331QZLUYeBPcgGwAtgvyQbgrVV1dlftPRyYuSupD50F/qo6eRvHl3bVtiRpZmbuLiJDZ+6aqSuNQ5dDPUuADwNPBB4EzqqqDyT5O+Cw9rR9gW9V1bKu+iFJeqgun/g3A2+sqhuS7A2sTrKyqn5tywlJ3gvc3WEfJElTdDnGvxHY2G7fm2Q9zRKL/wqQJMCvAsd11QdJ0tZ6mcefZClwFHDdxO5nA3dW1a0zXGPmriR1oPPAn2Qv4CLg9Kq6Z+LQycAFM11n5q4kdaPTWT1JdqMJ+udX1cUT+3cFXgI8q8v2JUlb67Ief4CzgfVV9b4ph58P3FJVG7pqX5I0vS6f+I8FTgHWJlnT7ntTu+ziy5hlmGeszNyV1IcuZ/V8FsgMx17RVbuSpNmZubuImLkrqQ+9rrmb5N1JbklyU5JLkuzbVfuSpOn1vebuSuDwqjoC+BLwRx22L0maRq9r7lbVVVW1uX17Lc1iLJKkHvW65u4UrwQ+OdNBM3clqRuDBP4kb6Yp4nb+TOeYuStJ3eh9Vk+SU4ETgOdVVfXdviSNXa+BP8nxwB8CP1tV3+mzbUlSo9c1d2lm8TwKWNlUdODaqnpNV33Y2Zi5K6kPfa+562LrkjQwM3cXETN3JfWhy8zdJUk+nWR9kpuTvKHdb/auJA2oy+mcW9bc/XHgGOB1SZ6O2buSNKguM3c3VtUN7fa9wHrgILN3JWlYQ665C9vI3pUk7XiDrbm7rexdSzZIUjc6DfyzrLm7JXv35TNl71qyQZK60WUC17Rr7pq9K0nD6n3NXeDPMXtXkgYzxJq7V3TV5s7Okg2S+jBkPX5J0gAs2bCI9FGywbIMkros2bB7ks8nubEt2fD2dv/jkqxMcmv787Fd9UGStLUuh3ruB46rqiOBZcDxSY4BzgQ+VVWHAp9q30uSetJlyYaqqm+3b3drXwWcBJzX7j8PeFFXfZAkba3rBK5d2qmcm4CVVXUd8ISq2ghNPR/ggBmuNXNXkjrQaeCvqgeqahlNIbajkxy+HdeauStJHehlOmdVfQtYBRwP3JnkQID256Y++iBJanQ5q2f/LYusJNkDeD5wC/Bx4NT2tFOBS7vqgyRpa13O4z8QOC/JLjS/YC6sqsuSfA64MMmrgNuBl3bYh52KmbuS+tBlyYabaGrwT93/78DzumpXkjQ7M3cXETN3JfVhiMXWlyW5Nsmadrrm0V31QZK0tS6f+Lcstn5Dkr2B1UlWAn8KvL2qPpnkBe37FR32Q5I0ocsx/o3AlkSte5OsBw6iyd59THvaPsAdXfVBkrS1Xsb4pyy2fjpwZZL30Aw1/fQM15wGnAZwyCGH9NFNSRqFIRZbfy1wRlUtAc6gWZ5xK2buSlI3hlhs/VRgy/bfA365K0k96nJWz7SLrdOM6f9su30ccGtXfZAkbW2IxdZfDXwgya7A92jH8WXmrqR+DLHYOsCzumpXkjQ7M3cXka4yd83WlTRpiDV3X9q+fzDJ8q7alyRNr8sn/i1r7n67nd3z2SSfBNYBLwH+qsO2JUkz6HKMv4Ct1tytqvUAzaQfSVLfhlhzd67XuuauJHXANXclaWSGWHNXkjSgIdbclSQNaIg1d18MfBDYH7g8yZqq+oUO+7HTMHNXUh+GWHP3EuCSrtqVJM3OzN1FxMxdSX0YInP33UluSXJTkku2fA8gSepHl7N6tmTuHgksA45PcgywEji8qo4AvgT8UYd9kCRN0Vngr8Z0mbtXVdXmdv+1NHP8JUk9GTpz95XAJ2e41sxdSerAYJm7Sd4MbAbOn+FaM3clqQODZO4mORU4AXh5W8xNktST3jN3kxwP/CFwYlV9p6v2JUnTGyJz98vAo4CVbWnma6vqNR32Q5I0YYjM3ad21ebOzpINkvrQyxi/JGnxsGTDIrI9JRsswyBpvgZ54k9yRlvGYV2SC5LsPkQ/JGmMeg/8SQ4CfgdYXlWHA7sAL+u7H5I0VkON8e8K7JFkV+DRwB0D9UOSRqf3wF9VXwfeA9wObATurqqrpp5nyQZJ6sYQQz2PBU4CngT8GLBnkt+Yep4lGySpG0MM9Twf+GpV3VVVPwAuBn56gH5I0igNEfhvB45J8ug0qbvPA9YP0A9JGqXe5/FX1XVJPgrcQFOd8wvAWX33YzEyc1dSHwZJ4KqqtwJvHaJtSRo7M3cXETN3JfVhkMCf5DbgXuABYHNVLR+iH5I0RkM+8T+3qr4xYPuSNEpW55SkkRkq8BdwVZLVSU6b7gQzdyWpG0MF/mOr6pnALwKvS/KcqSeYuStJ3Zg18Cd5bpKL2xLKNyf5aJIVC220qu5of24CLgGOXuhnSpLmZsbAn+SFwDnAJ4BfB14OXAGck+QF820wyZ5J9t6yDfw8sG6+nydJ2j6zzer5feBFVXXjxL41Sa4HPkjzS2A+ngBc0i60vivwkar6h3l+1sOKmbuS+jBb4H/ilKAPNIuoJ3nCfBusqq8AR873eknSwswW+O+b5zHN02yZu2bqStpRZgv8T0ny8Wn2B3jytj44yRLgw8ATgQeBs6rqA0mOBP4S2Au4DXh5Vd2zvR2XJM3PbIH/pFmOvWcOn70ZeGNV3dB+mbs6yUrgQ8DvVdXVSV5J813Cn8y5x5KkBZkx8FfV1Qv54KraSLO0IlV1b5L1wEHAYcA17WkrgSsx8EtSb2YM/EnW0mTYTquqjphrI0mWAkcB19FM3TwRuBR4KbBkhmtOA04DOOSQQ+balCRpG2Yb6jlhRzSQZC/gIuD0qrqnHd758yRvAT4OfH+666rqLNoFWpYvXz7jLyBJ0vaZbajnawv98CS70QT986vq4vZzb6FJ2iLJ0wCnq0hSjzqr1dOup3s2sL6q3jex/4D25yOAP6aZ4SNJ6kmX9fiPBU4B1iZZ0+57E3Bokte17y8G/qbDPuxUzNyV1Ic5Bf4kewCHVNUX5/rBVfVZmjn/0/nAXD9HkrRjbTPwJ/klmnn7jwSelGQZ8H9V1Ykd9210pmbumq0rqQtzGeN/G03Z5G8BVNUaYOm2LkpyTpJNSdZN7HtckpVJbm1/PnY+nZYkzd9cAv/mqrp7Hp99LnD8lH1nAp+qqkOBT7XvJUk9mkvgX5fk14Fdkhya5IPAP2/roqq6BvjmlN0nAee12+cBL9qOvkqSdoC5BP7fBn4CuB/4CHA3cPo823tCW8phS0mHA2Y60TV3Jakbs365m2QX4ONV9Xzgzf10qWHmriR1Y9Yn/qp6APhOkn12UHt3JjkQoP25aQd9riRpjuYyj/97NElYK5lYgKWqfmce7X0cOBV4V/vz0nl8hiRpAeYS+C9vX9slyQXACmC/JBuAt9IE/AuTvAq4naY6pySpR9sM/FV13rbOmeG6k2c49Lz5fN4YWLJBUh/mkrn7Vaapy19V21x+UZK0+MxlqGf5xPbuNMMzj+umO5Kkrm1zHn9V/fvE6+tV9WfAcdu6boaSDe9IclOSNUmuSvJjC+u+JGl7bTPwJ3nmxGt5ktcAe8/hs89l65IN766qI6pqGXAZ8Jbt7bAkaWHmMtTz3ontzcBXgV/d1kVVdU271u7kvnsm3u7JLGv6SpK6MZfA/6qq+srkjiRPmm+DSd4J/CZN6YfnznKei61LUgfmUqvno3PcNydV9eaqWgKcD7x+lvPOqqrlVbV8//33n29zkqQpZnziT/IfaYqz7ZPkJROHHkMzu2ehPkKTGPbWHfBZkqQ5mm2o5zDgBGBf4Jcm9t8LvHo+jSU5tKpubd+eCNwyn8+RJM3fjIG/qi4FLk3yU1X1ue394BlKNrwgyWHAg8DXgNfMq9eSpHmby5e7X0jyOpphnx8O8VTVK2e7aIaSDWdvX/ckSTvaXL7c/e/AE4FfAK4GDqYZ7pEk7YTmEvifWlV/AtzXFmx7IfCMhTQ6XVavJKkfcwn8P2h/fivJ4cA+wNIFtnsuW2f1SpJ6MJcx/rOSPBb4E5qFVPZigaUWpsvqlST1Yy71+D/Ubl4N9FaK2cxdSerGXIq0PSHJ2Uk+2b5/eruCVqfM3JWkbsxljP9c4EpgSwnlLwGnd9QfSVLH5hL496uqC2mSrqiqzcADnfZKktSZuQT++5I8nraEcpJjaCprzlub1fs54LAkG/oYOpIkNeYyq+d3aWbzPCXJPwH7A7+ykEZnWYhdktSx2apzHlJVt1fVDUl+lqZoW4AvVtUPZrpO22fpmZf/cPvc4/ccsCeSxmK2oZ6PTWz/XVXdXFXr5hr0kyxJ8ukk65PcnOQNE8d+O8kX2/1/Os++S5LmYbahnkxsz2f+/mbgje2/GPYGVidZCTwBOAk4oqruT3LAPD5bkjRPswX+mmF7TqpqI7Cx3b43yXrgIJpa/u+qqvvbY5u297MlSfM321DPkUnuSXIvcES7fU+Se5PcM8t1W2nLMxwFXAc8DXh2kuuSXJ3kJ2e45rQk1ye5/q677tqe5iRJs5htIZZddkQDSfYCLgJOr6p7kuwKPBY4BvhJ4MIkT66qh/yroqrOAs4CWL58+Xb/i0OSNL25zOOftyS70QT986vq4nb3BuDianyeJjFsvy77IUn6kc4Cf5LQrLi1vqreN3HoY8Bx7TlPAx4JfKOrfkiSHmouCVzzdSxwCrA2yZp235uAc4Bz2kVYvg+cOnWYZ0xue9cLf7i9atWq4ToiaTQ6C/xV9VkeOiV00m901a4kaXZdPvFrDszcldS3Tr/cnU6S3ZN8PsmNbebu2/vugySN2RBP/PcDx1XVt9tZP59N8smqunaAvkjS6PQe+Nsvcr/dvt2tfY32y11J6lvvQz0ASXZpZ/psAlZW1XXTnGPmriR1YJDAX1UPVNUy4GDg6CSHT3OOa+5KUgcGCfxbVNW3gFXA8UP2Q5LGZIhZPfsn2bfd3gN4PnBL3/2QpLEaYlbPgcB5SXah+cVzYVVdNkA/FgUzdyX1bYhZPTfRlGiWJA1g0DF+SVL/LNnQs8kSDVNZskFSH7osy3xOkk1tFc7J/S60LkkD6nKo51ymTNNM8lx+tND6TwDv6bB9SdI0Ogv8VXUN8M0pu1+LC61L0qD6/nJ3TgutgyUbJKkrfQf+yYXWf59mofVpF2uxZIMkdaPvwO9C65I0sL4D/8dwoXVJGlRn8/iTXACsAPZLsgF4Ky60/pASDVNZskFSH7pcbP3kGQ650LokDcjM3Z6ZuStpaL1m7iZ5W5KvJ1nTvl7QVfuSpOn1mrnben9VLWtfV3TYviRpGn1n7kqSBjZEWebXJ7mpHQp67EwnmbkrSd3oO/D/BfAUYBmwEXjvTCeauStJ3eg18FfVnVX1QFU9CPw1cHSf7UuSeg78SQ6cePtiYN1M50qSutF35u6KJMuAAm4Dfqur9hcrM3clDa3vzN2zu2pPkjQ3Zu72zMxdSUPrO3P37yaydm9Lsqar9iVJ0+vyif9c4L8AH96yo6p+bct2kvcCd3fYviRpGl2O8V+TZOl0x9pVt36Vtja/JKk/Q2TuAjwbuLOqbp3pBDN3JakbQwX+k4ELZjvBzF1J6kbvs3qS7Aq8BHhW321LkoZ54n8+cEtVbRigbUkavV4zd6vqbOBlbGOY5+HMzF1JQ+t9zd2qekVXbUqSts3M3Y7Nlqk7lZm7kvrQd+busiTXtpm71yexLLMk9azvNXf/FHh7VS0D3tK+lyT1qO81dwt4TLu9D3BHV+1LkqbX9xj/6cCVSd5D80vnp2c6MclpwGkAhxxySC+dk6Qx6Hse/2uBM6pqCXAGs9TnN3NXkrrRd+A/Fbi43f57XHNXknrXd+C/A/jZdvs4YMYibZKkbvS95u6rgQ+09Xq+RzuG/3A2W6buVGbuSupD75m7WJxNkgZl5m7HzNyVtNh0mbm7JMmnk6xPcnOSN7T735bk6xNr776gqz5IkrbW5RP/ZuCNVXVDkr2B1UlWtsfeX1Xv6bBtSdIMuhzj3whsbLfvTbIeOKir9iRJc9PLdM520fWjgOvaXa9PclNbyO2xffRBktToPPAn2Qu4CDi9qu4B/gJ4CrCM5l8E753hOhdbl6QOdBr4k+xGE/TPr6qLAarqzqp6oKoeBP6aGbJ3LdkgSd3oclZPaGrxrK+q903sP3DitBcD66ZeK0nqTpezeo4FTgHWJlnT7nsTcHKSZTQlmm8DfqvDPkiSpuhyVs9ngUxz6Iqu2lyMLNkgabHpu0ibJGlglmzomCUbJC02Q5RseEc7h39NkquS/FhXfZAkba3LoZ4tJRt+HDgGeF2SpwPvrqoj2gXXL6NZdF2S1JMuF1vfWFU3tNv3AuuBg9okri32pJndI0nqSS9j/FNLNiR5J/CbwN3Ac2e4xsXWJakDQ5RsoKre3C64fj7w+umuM3NXkrrRe8mGKT4C/HKXfZAkPdQQJRsOnTjtROCWrvogSdraECUbXpXkMOBB4GvAazrsw+DM3JW02FiyQZJGxszdDmxPtu4kM3cl9WGIzN3HJVmZ5Nb2pytwSVKPhsjcPRP4VFUdCnyqfS9J6knvmbvAScB57WnnAS/qqg+SpK0Nsdj6E6pqIzS/HIADZrjGNXclqQODZO7OhZm7ktSNITJ379yy7m77c1OXfZAkPVTvmbvAx4FT2+1TgUu76oMkaWtDZO6+C7gwyauA24GXdtiHQWxPtu4kM3cl9WGIzF2A53XVriRpdmbu7gDzzdSdysxdSX3ocox/9ySfT3Jjm7n79na/a+5K0oC6nNVzP3BcVR0JLAOOT3IMrrkrSYPqcoy/gG+3b3drX+Wau5I0rE7H+JPsAqwGngr816pyzV1JGlinCVxV9UA7pHMwcHSSw9v9rrkrSQPppVZPVX0LWAUcP+WQa+5KUs+6nNWzf5J92+09gOcDt7jmriQNq8sx/gOB89px/kcAF1bVZUkuerituTvfTN2pzNyV1IcuZ/XcRFOKeep+h3YkaUBm7s7TjsrWnWTmrqQ+dDnGf06STUnWTex7aZvF+2CS5V21LUmaWZezes5l61k864CXANd02K4kaRZdjvFf0y65OLlvPUBTql+SNIRe5vHPh2vuSlI3Fm3gN3NXkrqxaAO/JKkbBn5JGpkup3NeAHwOOCzJhiSvSvLiJBuAnwIuT3JlV+1LkqbX5ayek2c4dElXbfZpR5VpmGTJBkl9cKhHkkbGkg0TuijDsD0s2SCpD4M88Sd5Q5J1bfmG04fogySNVe+Bv12F69XA0cCRwAlTavRLkjo0xBP/jwPXVtV3qmozcDXw4gH6IUmjNETgXwc8J8njkzwaeAGwZOpJlmyQpG70HvjbQm3/L7AS+AfgRmDzNOdZskGSOjDIl7tVdXZVPbOqngN8E7h1iH5I0hgNMp0zyQFVtSnJITT1+X9qiH5I0hgNNY//oiSPB34AvK6q/v+B+vEQXWTjbg8zdyX1YZDAX1XPHqJdSZKZuw9h5q6kMRgigeuwJGsmXveYvStJ/en9ib+qvggsA0iyC/B1HiYVOyVpZzB0dc7nAf+zqr42cD8kaTSGDvwvAy6Y7oCZu5LUjcECf5JHAicCfz/dcTN3JakbQz7x/yJwQ1XdOWAfJGl0hgz8JzPDMI8kqTtDlWx4NPBzwG8N0f5MzNyVNAZDZe5+B3j8EG1L0tiNOnN36EzdqczcldSHzsb4kyxJ8ukk69u1dd/Q7n9p+/7BJMu7al+SNL0un/g3A2+sqhuS7A2sTrKSZgWulwB/1WHbkqQZdBb4q2ojsLHdvjfJeuCgqloJkKSrpiVJs+hlOmeSpcBRwHXbcY2Zu5LUgc4Df5K9gIuA06vqnrleZ+auJHWj08CfZDeaoH9+VV3cZVuSpLnpclZPgLOB9VX1vq7akSRtny5n9RwLnAKsTbKm3fcm4FHAB4H9gcuTrKmqX+iwHzMaOlN3KjN3JfWhy1k9nwVmmrrjwiuSNJCHfebuYsvOnY2Zu5L60OUY/zlJNiVZN7HvyCSfS7I2ySeSPKar9iVJ0+tyVs+5wPFT9n0IOLOqnkEz3PP7HbYvSZpGZ4G/qq4Bvjll92HANe32SuCXu2pfkjS9vhdiWUez3CLAS4ElM51o5q4kdaPvwP9K4HVJVgN7A9+f6UQzdyWpG73O6qmqW4CfB0jyNGBxTaSXpBHo9Yk/yQHtz0cAfwz8ZZ/tS5K6nc55AfA54LAkG5K8Cjg5yZeAW4A7gL/pqn1J0vS6zNw9eYZDH+iqzekstrIMs7Fkg6Q+9P3lriRpYAZ+SRoZA78kjYyBX5JGxsAvSSNj4JekkTHwS9LIGPglaWQM/JI0MgZ+SRoZA78kjYyBX5JGxsAvSSNj4JekkTHwS9LIGPglaWRSVUP3YZuS3AV8beh+9GA/4BtDd2IgY733sd43eO993Pt/qKr9p+7cKQL/WCS5vqqWD92PIYz13sd63+C9D3nvDvVI0sgY+CVpZAz8i8tZQ3dgQGO997HeN3jvg3GMX5JGxid+SRoZA78kjYyBvydJjk/yxSRfTnLmNMeT5M/b4zcleeZcr13MFnjf5yTZlGRdv73eMeZ770mWJPl0kvVJbk7yhv57vzALuPfdk3w+yY3tvb+9/97P30L+e2+P75LkC0ku67SjVeWr4xewC/A/gScDjwRuBJ4+5ZwXAJ8EAhwDXDfXaxfrayH33R57DvBMYN3Q99Lz3/mBwDPb7b2BL+0sf+c74N4D7NVu7wZcBxwz9D11fd8Tx38X+AhwWZd99Ym/H0cDX66qr1TV94G/BU6acs5JwIercS2wb5ID53jtYrWQ+6aqrgG+2WuPd5x533tVbayqGwCq6l5gPXBQn51foIXce1XVt9tzdmtfO8sMlAX9957kYOCFwIe67qiBvx8HAf828X4DW/+PPNM5c7l2sVrIfe/sdsi9J1kKHEXz5LuzWNC9t8Mda4BNwMqq2lnufaF/538G/AHwYEf9+yEDfz8yzb6pTzEznTOXaxerhdz3zm7B955kL+Ai4PSqumcH9q1rC7r3qnqgqpYBBwNHJzl8x3avM/O+7yQnAJuqavWO79bWDPz92AAsmXh/MHDHHM+Zy7WL1ULue2e3oHtPshtN0D+/qi7usJ9d2CF/71X1LWAVcPwO72E3FnLfxwInJrmNZojouCT/o7OeDv2FyBhewK7AV4An8aMvfX5iyjkv5KFf+nx+rtcu1tdC7nvi+FJ2zi93F/J3HuDDwJ8NfR8D3Pv+wL7t9h7AZ4AThr6nru97yjkr6PjL3V3n+PtBC1BVm5O8HriS5pv/c6rq5iSvaY//JXAFzTf+Xwa+A/yn2a4d4Da220LuGyDJBTT/E+yXZAPw1qo6u9+7mJ8F3vuxwCnA2nasG+BNVXVFj7cwbwu89wOB85LsQjMicWFVdTu1cQdZ6H/vfbJkgySNjGP8kjQyBn5JGhkDvySNjIFfkkbGwC9JI2Pg14ySPJBkTZJ1Sf4+yaMXQZ+WJvn1iffLk/z5Dvrs25LstyM+a47t7Zvk/+yrvWnaT5J/TPKY9s91qyqoSVYkqSS/NLHvsiQr2u1VSa6fOLY8yap2+xlJzu36PrT9DPyazXerallVHQ58H3jNXC5K0mV+yFLgh4G/qq6vqt/psL1OtPPU9wUGC/w088lvrG2Xg9gAvHmW4wck+cWpO6tqLXBwkkMW0Ed1wMCvufoM8NQke7Z18v+lrRt+EkCSV7T/KvgEcFX7/mNJPpHkq0len+R322uuTfK49rpVSZa32/u1Ketbnuw/k+SG9vXTbT/eBTy7/ZfIGe0T6WXtNY9r27ypbeOIdv/b2j6vSvKVJLP+omjbviXJh9p/7Zyf5PlJ/inJrUmOnvjc/94+Nd+a5NXt/iR5d3vt2iS/1u5fkabO/keAte29PKW9l3cn2SvJp9r7XTvxZ7s0TW3+v05To/6qJHu0x56a5P9LU7/+hiRPaff/fvt3dFNmrmn/cuDSae7/ye3f00+2u24E7k7yczN8zruBP57h2CeAl832560BDJ3m7GvxvoBvtz93pQkQrwX+M/Ab7f59aWrF7wm8gubJ8HHtsVfQZCfuTZOGfzfwmvbY+2kKj0FTi2V5u70fcFu7/Whg93b7UOD6dnsFE+nsk++BD9Jk9wIcB6xpt98G/DPwqLaNfwd2m+Z+b2uPLwU2A8+geThaDZxDk2Z/EvCxic+9kaa0wH40VRd/DPhlYCVN9uYTgNtpMlJXAPcBT2qvX8pEOYr2z/kxE38WX27b3NKfZe2xCyf+Dq4DXtxu797+uf08zWLeaft/GfCcae73a8Dek30BDgO+MNHWivb6ZwNXt/suA1ZM/v0B/wg8t91eNdHGscAnhv5v2ddDX5Zs0Gz2yI9KBnwGOJsmgJ6Y5Pfa/bsDW/4pv7KqJuvnf7qaevL3Jrmb5ukPmqfdI7bR9m7Af0myDHgAeNoc+vszNEGXqvrHJI9Psk977PKquh+4P8kmmoC8YZbP+mo1QxUkuRn4VFVVkrU0QXKLS6vqu8B3k3yapib7zwAXVNUDwJ1JrgZ+EriHpjbLV2doM8B/TvIcmtK8B7X93NKfNe32amBpkr2Bg6rqkvaev9f29+dpgv8X2vP3ovnlec2U9h7X/v1ssT/NL/hfrillQarqM0lI8uwZ+v5/0zz1/+GU/ZtofhlqETHwazbfraY87g8lCU1g+OKU/f8HzdPspPsnth+ceP8gP/pvbzM/GnLcfeL8M4A7gSPb49+bQ39nK4s72ZcH2PZ/+3Pp++TnT76frh9bTP0zmvRymuD7rKr6QTvsteXPZGr/95ilnQD/T1X91SxtAWxO8oiq2lL//W6af7UcC0xXD+qdNGP9m6ceaH/RvoOm8Nik3YHvbqMf6plj/NpeVwK/3f4CIMlRC/y824Bntdu/MrF/H2BjG5ROoRk2AbiXZvhoOtfQBE/SzDr5RnVfx/6kNOvEPp5mWORf2n78WpoFRfanWULy89NcO/Ve9qGpyf6DJM8F/sNsDbf3tiHJiwCSPCrNzKsrgVemqedPkoOSHDDNR3yRZpnALb4PvAj4zUzMnJpo7yrgsTS/jKfzTpqFRCY9jWYISYuIgV/b6x00wzA3pZn+944Fft57gNcm+Weace0t/htwapJraYLHliflm2ieVG9McsaUz3obsDzJTTRfnJ66wL7NxeeBy4FrgXdU1R3AJW0/b6QZ+/6DqvpfUy+sqn8H/qn9EvjdwPlt/6+n+QV2yxzaPwX4nfae/xl4YhugPwJ8rh2a+ijT/7K8nOaX1WSf7gNOAM7Y8uXyFO+kqSG/lWqqh941Zfdz23a0iFidU5qnJG+j+QL8PUP3ZT7SrPX64aqaabbOQj//UcDVwM9U1VbDQxqOT/zSSFXVRuCvkzymoyYOAc406C8+PvFL0sj4xC9JI2Pgl6SRMfBL0sgY+CVpZAz8kjQy/xsXV/FNXakOwQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sorted_idx = knn_perm.importances_mean.argsort()\n",
    "fig, ax = plt.subplots(figsize=(6, 7))\n",
    "ax.barh(ion_data.columns[sorted_idx], knn_perm.importances_mean[sorted_idx], zorder=1)\n",
    "ax.xaxis.grid(True)\n",
    "ax.set_ylabel(\"Feature ID\")\n",
    "ax.set_xlabel(\"Permuation Importance (kNN)\");\n",
    "ax.figure.savefig(\"PIscores.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "630ba84e",
   "metadata": {},
   "source": [
    "Build a mask representing features with an FI score > 0.  \n",
    "Create a reduced version of the dataset with just these features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "673cbf5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = knn_perm.importances_mean[sorted_idx]>0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fdbade71",
   "metadata": {},
   "outputs": [],
   "source": [
    "PI_features = ion_data.columns[sorted_idx][mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "328a9069",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Permutation Importance selects 18 features.\n"
     ]
    }
   ],
   "source": [
    "X_PI = X[PI_features]\n",
    "print(\"Permutation Importance selects\", X_PI.shape[1], \"features.\")\n",
    "X_PI_scaled = preprocessing.MinMaxScaler().fit_transform(X_PI)\n",
    "X_PI_train, X_PI_test, y_PI_train, y_PI_test = train_test_split(X_PI_scaled, \n",
    "                                                                   y, random_state=2, \n",
    "                                                                   test_size=1/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a156f39e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_Val on training PI features: 0.880\n",
      "Hold Out testing PI features: 0.847\n"
     ]
    }
   ],
   "source": [
    "knnPI = KNeighborsClassifier(n_neighbors=3)\n",
    "knnPI = knnPI.fit(X_PI_train,y_PI_train)\n",
    "\n",
    "y_pred_PI = knnPI.predict(X_PI_test)\n",
    "acc_PI = accuracy_score(y_pred_PI,y_test)\n",
    "cv_acc_PI = cross_val_score(knnPI, X_PI_train, y_train, cv=8)\n",
    "\n",
    "res_df.loc['After PI']['X-Val']=cv_acc_PI.mean()\n",
    "res_df.loc['After PI']['Hold-Out']=acc_PI\n",
    "\n",
    "\n",
    "print(\"X_Val on training PI features: {0:.3f}\".format(cv_acc_PI.mean())) \n",
    "print(\"Hold Out testing PI features: {0:.3f}\".format(acc_PI)) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a054d6f0",
   "metadata": {},
   "source": [
    "## Wrapper stage  \n",
    "Run the Wrapper process using feature backward elimination.  \n",
    "The `transform` method of the Wrapper object (`sfs`) will transform the dataset to include just the features selected by the Wrapper. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2666f71b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SequentialFeatureSelector\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "knn = KNeighborsClassifier(n_neighbors=3)\n",
    "sfs = SequentialFeatureSelector(knn, n_features_to_select='auto', direction = 'backward',\n",
    "                                cv = 10, tol = 0.00001\n",
    "                               )\n",
    "sfs.fit(X_PI_train, y_train)\n",
    "X_PI_W_train = sfs.transform(X_PI_train)\n",
    "X_PI_W_test = sfs.transform(X_PI_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8d6aa023",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(175, 16)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.8465909090909091"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knnPIW = KNeighborsClassifier(n_neighbors=3)\n",
    "knnPIW = knnPIW.fit(X_PI_W_train,y_PI_train)\n",
    "print(X_PI_W_train.shape)\n",
    "knnPIW.score(X_PI_W_test,y_PI_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "856da3aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(175, 16)\n",
      "X_Val on training all features: 0.897\n",
      "Hold Out testing all features: 0.847\n"
     ]
    }
   ],
   "source": [
    "knnPIW = KNeighborsClassifier(n_neighbors=3)\n",
    "knnPIW = knnPIW.fit(X_PI_W_train,y_PI_train)\n",
    "print(X_PI_W_train.shape)\n",
    "\n",
    "y_pred_PIW = knnPIW.predict(X_PI_W_test)\n",
    "acc_PIW = accuracy_score(y_pred_PIW,y_PI_test)\n",
    "cv_acc_PIW = cross_val_score(knnPIW, X_PI_W_train, y_train, cv=8)\n",
    "\n",
    "res_df.loc['After Wrapper']['X-Val']=cv_acc_PIW.mean()\n",
    "res_df.loc['After Wrapper']['Hold-Out']=acc_PIW\n",
    "\n",
    "\n",
    "print(\"X_Val on training all features: {0:.3f}\".format(cv_acc_PIW.mean())) \n",
    "print(\"Hold Out testing all features: {0:.3f}\".format(acc_PIW)) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23b472a2",
   "metadata": {},
   "source": [
    "## Plot all the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "4e2bf3f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAnz0lEQVR4nO3de5xVdb3/8ddbLg4CamqNCiqTefJSMOoIWpaDVzQVzQtgiVpG1NHzs6TCOiHqOV7rZKVG1FGyY+IlMVS8pZGXNAEdL6gUiemImnhBULnp5/fHWoObzZ6ZPcOsPTDr/Xw89oO91vqu9f2u/WXPZ6/vd63vVxGBmZnl10adXQAzM+tcDgRmZjnnQGBmlnMOBGZmOedAYGaWcw4EZmY550BguSJpiqT/yuC4kyT9sKOPa1YJDgQ5I+l5Se9JWlrw2rYDjnlgR5WxjPw2l3SlpFckLZH0N0nfq2D+J0t6oHBdRIyNiPMyym9PSXPSuvqbpEPK3K9G0geSrsiiXNZ1OBDk0xER0afgtbAzCyOpext3+QnQB9gF2Aw4EvhHR5drPXIZcDuwKXAI0FjmfqOBN4GRkjbOqGwlSepWyfxs3TgQGACSNpP0v5JelvSSpP9q+jJL2lHSvZJel7RI0jWSNk+3/RbYHrgl/cX6XUn1khqLjr/6qkHSREk3Svo/SW8DJ7eUfwl7Ab+LiDcj4oOIeDYibizIa2dJd0t6Q9I8Sce3cN6HS2qQ9Jakv0gaWLBtO0k3SXotPffLJO0CTAL2Sc/3rTTtGk1Okr4maX5ahumFV12SQtJYSX+X9KakyyWphepZBfwzPdcFETG3hbSFRgP/CawEjig67+Hpeb8t6R+ShqXrt5B0laSFadluTtevdRWUnscnCs7/F5JmSHoHGCrpC5IeS/N4UdLEov33TT/zt9LtJ0vaS9KrhT8OJB0jqaHMc7b2iAi/cvQCngcOLLH+ZuCXQG/gY8AjwNfTbZ8ADgI2Bj4K3Adc2twxgXqgsbl8gYkkf5yOIvkx0qul/EuU9dfAXOAUYKeibb2BF9Nt3YE9gEXAbun2KcB/pe/3AP4FDAG6ASel5dw4XX6c5OqjN1AF7JvudzLwQFG+hcfdP81zj/RYPwfuK0gbwK3A5iRB9DVgWAt19mOSX/a7t6GePwcsBz6S5j+9YNtgYHFapxsB/YCd0223Adel+/UA9mvhnAP4RMH5LwY+mx6zKv1/8Ol0eSDwKnBUmn57YAkwKs1nS6A23fY0cGhBPtOAMzv7u9OVX51eAL8qXOHJH7qlwFvp62agOv2j0asg3SjgT80c4yjgsaJjtjUQFP5hbGv+vYDvA3NIAsr8pj8cwAjg/qL0vwTOTt9P4cM/2L8AzitKOw/YD9gn/QPdvUT+pf4oFh73f4GLC7b1Scs5IF0O0qCSLl8PjG/mXEcCjwLDSJqEdk/XHwTMaaGefw3cnL7fJ83/YwWfx09K7LMN8AHwkTLPuTgQXN3K/71Lm/IFzgKmNZPue8A16fstgHeBbTr7u9OVX21tm7Wu4aiI+GPTgqTBJL/KXi5oodiI5Jc1kj4G/IzkV2bfdNub61iGFwve79BS/sUi4j3gfOB8SZsC44EbJG2fHmtIU5NNqjvw2xKH2gE4SdLpBet6AtsC75M0x6xq43mR7v9oQXmXSnqd5Jf38+nqVwrSv0sSLEr5f8BlEXGHpLHAHWkzzmeAP5baQVIv4Djg1DT/hyS9AJxA8sd4O2BGiV23A96IiPbW7Rr1JWkIcCHwKZLPdWPghoK8muvX+T/gGUl9gONJAvvL7SyTlcF9BAbJF3g5sFVEbJ6+No2I3dLtF5D8+hsYEZsCXwYK27SLh7B9B9ikaSFt6/9oUZrCfVrLv1kR8TZJUOgN1KTH+nPBcTaPpEP8G82c938Xpd0kIq5Nt23fTEd2a0P2LiQJMgBI6k3S9PFSa+dTQneSPgIi4lbg28BdJL/Q/6eZfY4m6Vi+QsmdVa+QBKHR6fYXgR1L7PcisEVT/0+R4jrdukSa4s/ld8B0YLuI2Iykb6Xp/01zZSAiXgIeSs/jREoHcetADgRG+mvrLuDHkjaVtFHaQbxfmqQvaXOSpH7Ad4oO8Srw8YLlvwFVaWdhD5IOy2bvWikj/zVI+mHaqdhTUhXJr+a3SJp1bgX+TdKJknqkr73STt5ivwLGShqiRO+0zH1J+iheBi5M11dJ+mzB+faX1LOZU/odcIqkWiV365wP/DUinm/uM2jBDcAESYMkbUTy2b7Hh/0WpZwEXEnSPl+bvj4L1Er6NEnT1SmSDkg/636Sdk7r4XaSAPKR9LP7fHrMx4Hd0nOqImnea01fkiuMZelV5wkF264BDpR0vKTukraUVFuw/Wrgu+k5TCsjL1sHDgTWZDTJ5fvTJM0+N5K0GQOcQ9LxuZikM/Gmon0vAP4zvftjXEQsBr5J0k79EsmvydZueWwp/2IBXEXSIbuQpL38CxGxNCKWAAeTtK0vJGmCuYgSgSgiZgNfI7k9802SvoaT023vk9xp8wnghbT8I9Jd7yXprH5F0qISx70H+CHwe5JgsmNanvb4Eckf9WnAGyRNdGOB3wC3SdqsMHEaqA8g6cx/peA1B7gDOCkiHiHpTP8JSZ3+mQ+vYE4k6U94lqQj/Yz0nP4GnEvSHPV3YI07iJrxTeBcSUuACSR9IaTHewE4DDgzPa8GYFDBvtPSMk2LiHfKyMvWgSI8MY2ZrX8k/YPkzrGSfSHWcXxFYGbrHUnHkFz53dvZZckD3zVkZusVSTOBXYETI+KDTi5OLrhpyMws59w0ZGaWcxtc09BWW20VAwYM6OxiZOadd96hd+/enV0MayfX34arq9fdnDlzFkVE8fM8wAYYCAYMGMDs2bM7uxiZmTlzJvX19Z1dDGsn19+Gq6vXnaR/NrfNTUNmZjnnQGBmlnMOBGZmOedAYGaWcw4EZmY550BgZpZzDgRmZjnnQGBmlnMOBGZmOedAYGaWcw4EZmY550BgZpZzDgRmZjnnQGBmlnMOBGZmOedAYGaWcw4EZmY5l2kgkDRM0jxJ8yWNL7H9I5KmSXpC0iOSPpVleczMbG2ZBQJJ3YDLgUOBXYFRknYtSvZ9oCEiBgKjgZ9mVR4zMystyyuCwcD8iHguIlYAU4HhRWl2Be4BiIhngQGSqjMsk5mZFcly8vp+wIsFy43AkKI0jwNfBB6QNBjYAegPvFqYSNIYYAxAdXU1M2fOzKjInW/p0qVd+vy6OtffhivPdZdlIFCJdVG0fCHwU0kNwJPAY8CqtXaKmAxMBqirq4v6+voOLej6ZObMmXTl8+vqXH8dZ8D42yqa35RhfXJbd1kGgkZgu4Ll/sDCwgQR8TZwCoAkAQvSl5mZVUiWfQSzgJ0k1UjqCYwEphcmkLR5ug3gVOC+NDiYmVmFZHZFEBGrJJ0G3Al0A66MiLmSxqbbJwG7AFdLeh94GvhqVuUxM7PSsmwaIiJmADOK1k0qeP8QsFOWZbB8q3w7c++K5mfWEfxksZlZzjkQmJnlnAOBmVnOORCYmeWcA4GZWc45EJiZ5Vymt4+a5c7LDTCxeGzFDE1cXLm8uroc152vCMzMcs6BwMws5xwIzMxyzoHAzCznHAjMzHLOdw21woOWmVlXl+kVgaRhkuZJmi9pfIntm0m6RdLjkuZKOiXL8piZ2doyCwSSugGXA4eSTFI/StKuRcn+HXg6IgYB9cCPCyaqMTOzCsjyimAwMD8inouIFcBUoPhpjQD6ptNU9gHeoMScxWZmlh1FFM8n30EHlo4FhkXEqenyicCQiDitIE1fkukrdwb6AiMiYq1GeUljgDEA1dXVe06dOjWTMpfy5EuVffqvZrNu9OnTp6J5dmUVr79ey+mzfGHrCTvKNrWVy6vCXHcda+jQoXMioq7Utiw7i1ViXXHUOQRoAPYHdgTulnR/8bzFETEZmAxQV1cX9fX1HV7Y5pxc6c7iQX+nfvbZlctwPXrMPQuVr78F1M+rYP2N6rr157qrnCybhhqB7QqW+wPF4fYU4KZIzAcWkFwdmJlZhWQZCGYBO0mqSTuAR5I0AxV6ATgAQFI18EnguQzLZGZmRTJrGoqIVZJOA+4EugFXRsRcSWPT7ZOA84Apkp4kaUr6XkQsyqpMZma2tkwfKIuIGcCMonWTCt4vBA7OsgxmZtYyDzFhZpZzDgRmZjnnQGBmlnMOBGZmOedAYGaWcw4EZmY550BgZpZzDgRmZjnnQGBmlnMOBGZmOedAYGaWcw4EZmY519mT139HUkP6ekrS+5K2yLJMZma2pk6dvD4iLomI2oioBc4C/hwRb2RVJjMzW1tnT15faBRwbYblMTOzErIMBP2AFwuWG9N1a5G0CTAM+H2G5TEzsxI6e/L6JkcADzbXLCRpDDAGoLq6mpkzZ3ZIActx5qdXVSwvgKUbb8vMT55TuQwr+Fl2Btffhst1VzlZBoJyJq9vMpIWmoUiYjIwGaCuri7q6+s7qIitO3n8bRXLC2DKoAXUzzu7chmOWly5vDqB62/D5bqrnM6evB5JmwH7AX/IsCxmZtaMzp68HuBo4K6IeCerspiZWfM6dfL6dHkKMCXLcpiZWfP8ZLGZWc45EJiZ5ZwDgZlZzjkQmJnlnAOBmVnOORCYmeWcA4GZWc45EJiZ5ZwDgZlZzjkQmJnlnAOBmVnOORCYmeWcA4GZWc5lGggkDZM0T9J8SeObSVMvqUHSXEl/zrI8Zma2tsyGoZbUDbgcOIhktrJZkqZHxNMFaTYHrgCGRcQLkj6WVXnMzKy0LK8IBgPzI+K5iFgBTAWGF6U5AbgpIl4AiIh/ZVgeMzMrQRHNzSe/jgeWjiX5pX9qunwiMCQiTitIcynQA9gN6Av8NCKuLnGswsnr95w6dWomZS7lyZcqO69oTa/l9Fne3NTOGdimtnJ5dQLX34bLddexhg4dOici6kpty3KGMpVYVxx1ugN7AgcAvYCHJD0cEX9bYydPXp+d9WgC7Sy4/jZcrrvKabVpSNLhktrThNQIbFew3B8oDreNwB0R8U5ELALuAwa1Iy8zM2uncv7AjwT+LuliSbu04dizgJ0k1UjqmR5nelGaPwCfk9Rd0ibAEOCZNuRhZmbrqNWmoYj4sqRNgVHAVZICuAq4NiKWtLDfKkmnAXcC3YArI2KupLHp9kkR8YykO4AngA+AX0fEU+t+WmZmVq6y+ggi4m1Jvydpxz8DOBr4jqSfRcTPW9hvBjCjaN2kouVLgEvaWG4zM+sg5fQRHCFpGnAvyR0+gyPiUJK2/HEZl8/MzDJWzhXBccBPIuK+wpUR8a6kr2RTLDMzq5RyAsHZwMtNC5J6AdUR8XxE3JNZyczMrCLKuWvoBpKO3Cbvp+vMzKwLKCcQdE+HiAAgfd8zuyKZmVkllRMIXpN0ZNOCpOHAouyKZGZmlVROH8FY4BpJl5EMG/EiMDrTUpmZWcWU80DZP4C9JfUhGaSu2YfIzMxsw1PWA2WSvkAyQmiVlIwlFxHnZlguMzOrkHIeKJsEjABOJ2kaOg7YIeNymZlZhZTTWfyZiBgNvBkR5wD7sOaoomZmtgErJxAsS/99V9K2wEqgJrsimZlZJZXTR3BLOrfwJcCjJJPL/CrLQpmZWeW0eEWQTkhzT0S8FRG/J+kb2DkiJpRzcEnDJM2TNF/S+BLb6yUtltSQvso6rpmZdZwWrwgi4gNJPybpFyAilgPLyzmwpG7A5cBBJDORzZI0PSKeLkp6f0Qc3uaSm5lZhyinj+AuSceo6b7R8g0G5kfEc+mwFFOB4W0uoZmZZUoRxfPJFyWQlgC9gVUkHccCIiI2bWW/Y4FhEXFqunwiMCQiTitIUw/8nuSKYSEwLiLmljjWGGAMQHV19Z5Tp04t8/TW3ZMvVXaC6Zpey+mzvHhq5wxtU1u5vDqB62/D5brrWEOHDp0TEXWltpXzZHHfduZb6gqiOOo8CuwQEUslHQbcDOxUogyTgckAdXV1UV9f384itd3J42+rWF4AUwYtoH7e2ZXLcFRlv2yV5vrbcLnuKqfVQCDp86XWF09UU0Ijaz5v0J/kV3/hMd4ueD9D0hWStooID2pnZlYh5dw++p2C91Ukbf9zgP1b2W8WsJOkGuAlYCRwQmECSVsDr0ZESBpM0mfxepllNzOzDlBO09ARhcuStgMuLmO/VZJOA+4EugFXRsRcSWPT7ZOAY4FvSFoFvAeMjNY6LczMrEOVNehckUbgU+UkjIgZwIyidZMK3l8GXNaOMpiZWQcpp4/g53zYybsRUAs8nmGZzMysgsq5Iphd8H4VcG1EPJhReczMrMLKCQQ3Assi4n1InhiWtElEvJtt0czMrBLKebL4HqBXwXIv4I/ZFMfMzCqtnEBQFRFLmxbS95tkVyQzM6ukcgLBO5L2aFqQtCfJrZ5mZtYFlNNHcAZwg6Smp4K3IZm60szMuoByHiibJWln4JMk4wc9GxErMy+ZmZlVRDmT1/870DsinoqIJ4E+kr6ZfdHMzKwSyukj+FpEvNW0EBFvAl/LrERmZlZR5QSCjQonpUlnHuuZXZHMzKySyuksvhO4XtIkkqEmxgK3Z1oqMzOrmHICwfdIZgf7Bkln8WMkdw6ZmVkX0GrTUER8ADwMPAfUAQcAz5RzcEnDJM2TNF/S+BbS7SXp/XR6SzMzq6Bmrwgk/RvJZDKjSCaLuQ4gIoaWc+C0L+Fy4CCSoatnSZoeEU+XSHcRSROUmZlVWEtXBM+S/Po/IiL2jYifA++34diDgfkR8VxErACmAsNLpDudZAL7f7Xh2GZm1kFa6iM4huSK4E+S7iD5Q15qQvrm9ANeLFhuBIYUJpDUDziaZNrLvZo7kKQxJP0UVFdXM3PmzDYUY92c+elVFcsLYOnG2zLzk+dULsMKfpadwfW34XLdVU6zgSAipgHTJPUGjgK+BVRL+gUwLSLuauXYpYJG8TSUlwLfi4j3C+5QLVWWycBkgLq6uqivr28l645z8vjbKpYXwJRBC6ifd3blMhy1uHJ5dQLX34bLdVc55Qwx8Q5wDXCNpC2A44DxQGuBoBHYrmC5P7CwKE0dMDUNAlsBh0laFRE3l1V6MzNbZ22aszgi3gB+mb5aMwvYSVIN8BJJM9MJRceraXovaQpwq4OAmVlltWfy+rJExCpJp5HcDdQNuDIi5koam26f1OIBzMysIjILBAARMQOYUbSuZACIiJOzLIuZmZVWzlhDZmbWhTkQmJnlnAOBmVnOORCYmeWcA4GZWc45EJiZ5ZwDgZlZzjkQmJnlnAOBmVnOORCYmeWcA4GZWc45EJiZ5VymgaC1yeslDZf0hKQGSbMl7ZtleczMbG2ZjT5a5uT19wDTIyIkDQSuB3bOqkxmZra2LK8IWp28PiKWRkTT9JW9WXsqSzMzy1iWgaDU5PX9ihNJOlrSs8BtwFcyLI+ZmZWgD3+Qd/CBpeOAQyLi1HT5RGBwRJzeTPrPAxMi4sAS28YAYwCqq6v3nDp1aiZlLuXJlyo7wXRNr+X0WV48tXOGtqmtXF6dwPW34XLddayhQ4fOiYi6UtuynKGsnMnrV4uI+yTtKGmriFhUtG0yMBmgrq4u6uvrMyhuaSePv61ieQFMGbSA+nlnVy7DUZX9slWa62/D5bqrnCybhlZPXi+pJ8nk9dMLE0j6hCSl7/cAegKvZ1gmMzMr0tmT1x8DjJa0EngPGBFZtVWZmVlJnTp5fURcBFyUZRnMzKxlfrLYzCznHAjMzHLOgcDMLOccCMzMcs6BwMws5xwIzMxyzoHAzCznHAjMzHLOgcDMLOccCMzMcs6BwMws5xwIzMxyzoHAzCznMg0EkoZJmidpvqTxJbZ/SdIT6esvkgZlWR4zM1tbZoFAUjfgcuBQYFdglKRdi5ItAPaLiIHAeaSzkJmZWeVkeUUwGJgfEc9FxApgKjC8MEFE/CUi3kwXHyaZztLMzCooy8nrjwWGFU1ePyQiTmsm/Thg56b0Rds8eX1WuvDk5+D625C57jpWZ01erxLrSkYdSUOBrwL7ltruyesztB5NoJ0F19+Gy3VXOVkGgkZgu4Ll/sBa4VbSQODXwKER4YnrzcwqLMs+glnATpJqJPUERgLTCxNI2h64CTgxIv6WYVnMzKwZmV0RRMQqSacBdwLdgCsjYq6ksen2ScAEYEvgCkkAq5prwzIzs2xk2TRERMwAZhStm1Tw/lRgrc5hMzOrnEwDgZlZW61cuZLGxkYmH7k1KnnPSTY267kVzwy4vmL58cwzmRy2qqqK/v3706NHj7L3cSAws/VKY2Mjffv2pbr75qRNxhVR02s5fSt5++i2u3T4ISOC119/ncbGRmpqasrez2MNmdl6ZdmyZWy55ZYVDQJdhSS23HJLli1b1qb9HAjMbL3jINB+7fnsHAjMzHLOfQRmtl478rIHO/R400/7bKtpXnzpFT5/zKnMuf0atvjIZrz51tvsMewEZt44mR36b8uU66Zz558f4torLli9z6I33mSX/Y6hcfYdbLxxz7WOOeW66cx+4mku+++1BmLudL4iMDMrsl2/rfnG6GMZf8HPABh//s8Y86UvskP/bQH44mH7c/d9f+Xd995bvc+Nt97DkQfvVzIIrO8cCMzMSvjW177Ew48+yaW/uoYHZjVw5tdPXL1t0759+Pzeu3PLXfetXjd1+p2MGj6MW+76M0MOH83uB4/iwBFjefW19X/kHAcCM7MSevTowSX/eQbfmvhjLj1nHD17rnlf/qjhw5g6/S4AFr7yGn977p8M/Wwd+w7enYdv+Q2P3XUtI4cfwsVX/KYzit8m7iMwM2vG7fc+yDbVW/HUs/M56PN7r7Ht8AM/xze/fwFvL1nK9bfcxbFfOJBu3brR+PKrjPjGeF7+1yJWrFhJzfbbdlLpy+crAjOzEhqemsfd9/+Vh2/5DT/51e948aVXqD1oJLUHjWTS1TfSq1cVw+o/w7Tb/8TUP9zJqOGHAHD6Dy/mtFNG8OQ91/PLi37AsuUrOvlMWucrAjOzIhHBN846n0vPGcf2/bbhO98YzfgLfk7D3WtOijXqqGGcdcHPeXvpO+y950AAFr+9lH5bfxSA39xwS8XL3h4OBGa2Xivnds+O9qtrbmL7fluvbg765knHMeX66fz5oTnst8+eq9MdvN/enHTGBL466qjVD3JNPPPrHPf179Fv64+y9x6fZsGLFRy2op0yDQSShgE/JRmG+tcRcWHR9p2Bq4A9gB9ExI+yLI+ZWTnGfPkYxnz5mNXL3bp1Y84dv1srXffu3XntyXvXWDf8kHqGH1K/VtqTRxzJySOO7PCydoTMAoGkbsDlwEEks5XNkjQ9Ip4uSPYG8B/AUVmVw8zMWpZlZ/FgYH5EPBcRK4CpwPDCBBHxr4iYBazMsBxmZtaCLJuG+gEvFiw3AkPacyBJY4AxANXV1cycOXOdC1euMz+9qmJ5ASzdeFtmfvKcymVYwc+yM7j+NjybbbYZS5YsobpXZfN9f6MeLNm4grd6LlmS2aGXLVvWpr+TWQaCUkPgRXsOFBGTgckAdXV1UV9fvw7FapuTx99WsbwApgxaQP28syuX4ajFlcurE7j+NjzPPPMMffv2ZUHjWxXNt6bXysrOR7Dl7pkduqqqit13L//4WTYNNQLbFSz3B9b/7nMzs5zJMhDMAnaSVCOpJzASmJ5hfmZm1g6ZNQ1FxCpJpwF3ktw+emVEzJU0Nt0+SdLWwGxgU+ADSWcAu0bE21mVy8w2LAN/vUOHHu+JU//Zapo+O32WpX//cPjrcoaQnvjjSfTpvQnjxo5eY/3zLy7k8JP+H0/de0PJ/W6++WYmTJjAihUr6NGjB+eddx5HHXVUi+VraGhg4cKFHHbYYa2eSzkyfY4gImYAM4rWTSp4/wpJk5GZWe48/vjjjBs3jrvvvpuamhoWLFjAQQcdxMc//nEGDhzY7H4NDQ3Mnj27wwKBxxoyM2uDfzYu5IDjv87AA4/ngOO/zgsvvbxWmjlPPM2gA0ewzxEncfmU65o91o9+9CO+//3vr55ovqamhrPOOotLLrkEgPr6embPng3AokWLGDBgACtWrGDChAlcd9111NbWct11zR+/XA4EZmZF3lu2fPUAc7UHjWTCj1Y3ZHDaDy5i9LGH88Qfr+dLXzyU//jhJWvtf8q3J/Kz877LQ7e0PAT13Llz2XPPPddYV1dXx9y5c5vdp2fPnpx77rmMGDGChoYGRowY0baTK8GBwMysSK+qjWm4e+rq17njxq7e9tCcJznh6GEAnHjMF3jgkYY19l389hLeWrx09ZhEJx7zhWbziYi1JpsvtS5rDgRmZuug+G92xNrrmpzyrbOpPWgkh514OgC77bbb6qafJo8++ii77rorkIxl9MEHHwDJQ2JZcSAwM2uDz9QNZOof7gTgmptuZ9/Baz64tflmfdls0z488MhjSZppt6/edtVPzqHh7qnM+O3PARg3bhwXXHABzz//PADPP/88559/PmeeeSYAAwYMYM6cOQDceOONq4/Tt29flnTgk8kehtrM1mvl3O5ZST8777t85dsTuWTS1Xx0i49w1U8mrpXmqv+ZyFe+fQ6b9KrikPp9mj1WbW0tF110EUcccQQrV66kR48eXHzxxdTW1gJJoDj++OP57W9/y/777796v6FDh3LhhRdSW1vLWWedtc79BIpo16gPnaauri6KL6WyNKDiQxT8vbJDFEzsekMUFHL9bXieeeYZdtllF56o+BATyys7xMS22Q0x0fQZFpI0JyLqSqV305CZWc45EJiZ5ZwDgZmtdza0Juv1SXs+OwcCM1uvVFVV8frrrzsYtENE8Prrr1NVVdWm/XzXkJmtV/r3709jYyOvvvwGKjmtSTbe77mSqpVvVSw/Fj+TyWGrqqro379tQ7g5EJjZeqVHjx7U1NQw9JdPt564A00Z9Hd2z+kdX5k2DUkaJmmepPmS1hq/VYmfpdufkLRHluUxM7O1ZRYIJHUDLgcOBXYFRknatSjZocBO6WsM8IusymNmZqVleUUwGJgfEc9FxApgKjC8KM1w4OpIPAxsLmmbDMtkZmZFsuwj6Ae8WLDcCAwpI00/YI0BviWNIbliAFgqaV7HFnX9MRS2AhZVLMNzKjvKYVfn+ttw5aDump3qLctAUOosi+8HKycNETEZmNwRhVrfSZrd3GPgtv5z/W248lx3WTYNNQLbFSz3B4oH8ignjZmZZSjLQDAL2ElSjaSewEhgelGa6cDo9O6hvYHFEbH2vG9mZpaZzJqGImKVpNOAO4FuwJURMVfS2HT7JJKJ7Q8D5gPvAqdkVZ4NSC6awLow19+GK7d1t8ENQ21mZh3LYw2ZmeWcA4GZWc45EJRB0tGSQtLOBesGSHoqfV8v6dYS+9VLWiypIX39sZ35nyFpk/afgZWrmbr+qKS/SnpM0uckfbMD8pmZDr/yuKQHJX2yYH3ubmGsxOcuabikmwuWz5I0v2D5CEnFN7TkggNBeUYBD5Dc+dRW90dEbfo6sJ35nwG0KRBI8oCC7VOqrg8Ano2I3UkegGzTH6T0rrhS37UvRcQg4DfAJe0sb1dRic/9L0DhBML7AG9L+li6/BngwaJjVPx71Bl5OhC0QlIf4LPAV2lfICh1zC9LeiS9SvhlOi4Tkn4habakuZLOSdf9B7At8CdJf0rXLS041rGSpqTvp0j6nzTdRZJ2lHSHpDmS7m/6tSXpOElPpb9G7+uIc+oKStW1pFrgYuAwSQ3ARcCOad1dkqb5jqRZ6cCJTfU2QNIzkq4AHmXN52WK3Qd8IqPTWu9V6nOPiNeAxZKaPut+wO9JAgDpv3+RNFHSZEl3AVenx7xf0qPp6zNpXvWS7pM0TdLTkiY1BR5JSyX9OE1/j6SPpuub+06u8d3N4GNuWUT41cIL+DLwv+n7vwB7pO8HAE+l7+uBW0vsWw8sBhrS1w+AXYBbgB5pmiuA0en7LdJ/uwEzgYHp8vPAVgXHXVrw/lhgSvp+CnAr0C1dvgfYKX0/BLg3ff8k0C99v3lnf8bry6uFuj4ZuKy43tPlg0luOxTJD6tbgc+n6T4A9m4mr5lAXfr+O8B1xevz8qrw5z4FGA18kmT8swNIAk534E2gCpgIzAF6pftsAlSl73cCZqfv64FlwMfT7+zdwLHptiC54gOYUHAezX0n1/juVvrl5oPWjQIuTd9PTZcfbcP+90fE4U0LSp6t2BOYJQmgF/CvdPPxSsZV6g5sQzJq6xNtLO8NEfF++ivrM8ANaT4AG6f/PghMkXQ9cFMbj9+VtaeuD05fj6XLfUj+WLwA/DOSwRSbc42k90gC/entK3KXUMnP/UGS70U34CHgEZI/1LsD8yJiWfp9mR4R76X79AAuS69S3gf+reB4j0TEcwCSrgX2BW4kCUbXpWn+D7iple8kpN/dVs47Ew4ELZC0JbA/8ClJQfKfJyR9d10OC/wmIs4qyqsGGAfsFRFvps09zc03V/jwR3Gad9J/NwLeiojatXaOGCtpCPAFoEFSbUS83uYz6ULWoa4FXBARvyw63gA+rIvmfCkiZrezyF1CJ3zufyEJut2AX0XEEklVJL/uC/sHCo/xLeBVYBDJ92pZwbbiB7GaezAraOE7WSLPinIfQcuOJRkme4eIGBAR2wELSKJ+e90DHNvUQSVpC0k7AJuS/EdYLKmaZK6GJkuAvgXLr0raJW2PPLpUJhHxNrBA0nFpPpI0KH2/Y0T8NSImkIy22FL7dV6UW9fFdXEn8JX01x6S+hV0PlrrKv25P03S5/Y5PryaaADGkgSJUjYDXo6ID4ATSYJIk8FKhtHZCBhB0uENyd/WY9P3JwAPtPSd7GwOBC0bBUwrWvd7koptl4h4GvhP4C5JT5C0K24TEY+T/MecC1zJmr9OJgO3N3UWA+NJ2hPvpWjI7iJfAr4q6fH0uE3zQVwi6Uklt7/eBzze3vPpQsqq6/TK6cG0s/2SiLgL+B3wkKQnSZoF+mLlqujnHkmD/F+BRRGxMl39EEk7f3OB4ArgJEkPkzQLFf5yfwi4EHiKJIA1ncs7wG6S5pBc8Zybrm/uO9mpPMSEmVk7SKoHxhX2ARZsWxoRfSpeqHbyFYGZWc75isDMLOd8RWBmlnMOBGZmOedAYGaWcw4EZoCSkS9/W7DcXdJrKjGqbNF+tZIOK1ieKGncOpRjnfY3aw8HArPEOyRPt/ZKlw8CXipjv1qS6VbNNlgOBGYfup1k2A1IHnS6tmmDpN6SrkxHu3xMydj2PUkeFBqhZFTMEWnyXZXMK/CcktFjm47x7fSBqKcknVGw/gdK5ib4I8lgaGYV5UBg9qGpwMh07JmBJE+gNvkByUiRewFDSeYP6EEyYNl1kcw30TTI2M7AIcBg4GxJPSTtCZxCMuLk3sDXJO2erh9JMujZF4G9sj5Js2IedM4sFRFPpIOWjQJmFG0+GDiyoP2+Cti+mUPdFhHLgeWS/gVUk4ydMy0i3gGQdBPJeDcbpevfTdfncoYs61wOBGZrmg78iGQ0yi0L1gs4JiLmFSZOR3Ettrzg/fsk3zOVSNfET3Vap3LTkNmargTOjYgni9bfCZyudCB5Sbun64tHxWzOfcBRkjaR1Jtk1Nj70/VHS+olqS9wREechFlb+IrArEBENAI/LbHpPJLJU55Ig8HzwOHAn4DxSqZTvKCF4z6azjHxSLrq1xHxGICk60iGQv4nSXAwqyiPNWRmlnNuGjIzyzkHAjOznHMgMDPLOQcCM7OccyAwM8s5BwIzs5xzIDAzy7n/D5z78sPDjW2pAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = res_df.plot.bar(rot=0)\n",
    "ax.set_ylabel(\"Accuracy\")\n",
    "ax.set_title(\"Feature Selection & Accuracy\")\n",
    "ax.set_ylim(0,1)\n",
    "ax.set_yticks(np.arange(0,1,0.1))\n",
    "ax.legend(loc='lower right')\n",
    "ax.grid(True)\n",
    "ax.figure.savefig(\"PI+Wrapper.png\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}