notebooks/15 - Advanced Strategies (Schemas).ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Strategies (Schemas)\n",
    "\n",
    "Validating data to be within certain ranges is an advanced strategy when automating data science processes.\n",
    "\n",
    "In data science, automation has become an essential aspect of various processes. \n",
    "\n",
    "One of the critical challenges in automating data science workflows is ensuring the accuracy and validity of the data being used. Validating data to be within certain ranges is an advanced strategy that can be employed to ensure that the data being used is reliable and accurate. This approach involves setting predetermined limits or ranges for specific data points and verifying that the data falls within these parameters. \n",
    "\n",
    "By implementing this strategy, data scientists can improve the accuracy and reliability of their automated data science workflows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How To"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pandera as pa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-122.23</td>\n",
       "      <td>37.88</td>\n",
       "      <td>41.0</td>\n",
       "      <td>880.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>8.3252</td>\n",
       "      <td>452600.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-122.22</td>\n",
       "      <td>37.86</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7099.0</td>\n",
       "      <td>1106.0</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>8.3014</td>\n",
       "      <td>358500.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-122.24</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1467.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>7.2574</td>\n",
       "      <td>352100.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1274.0</td>\n",
       "      <td>235.0</td>\n",
       "      <td>558.0</td>\n",
       "      <td>219.0</td>\n",
       "      <td>5.6431</td>\n",
       "      <td>341300.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1627.0</td>\n",
       "      <td>280.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>259.0</td>\n",
       "      <td>3.8462</td>\n",
       "      <td>342200.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0    -122.23     37.88                41.0        880.0           129.0   \n",
       "1    -122.22     37.86                21.0       7099.0          1106.0   \n",
       "2    -122.24     37.85                52.0       1467.0           190.0   \n",
       "3    -122.25     37.85                52.0       1274.0           235.0   \n",
       "4    -122.25     37.85                52.0       1627.0           280.0   \n",
       "\n",
       "   population  households  median_income  median_house_value ocean_proximity  \n",
       "0       322.0       126.0         8.3252            452600.0        NEAR BAY  \n",
       "1      2401.0      1138.0         8.3014            358500.0        NEAR BAY  \n",
       "2       496.0       177.0         7.2574            352100.0        NEAR BAY  \n",
       "3       558.0       219.0         5.6431            341300.0        NEAR BAY  \n",
       "4       565.0       259.0         3.8462            342200.0        NEAR BAY  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/housing.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-122.23</td>\n",
       "      <td>37.88</td>\n",
       "      <td>41.0</td>\n",
       "      <td>880.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>8.3252</td>\n",
       "      <td>452600.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-122.22</td>\n",
       "      <td>37.86</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7099.0</td>\n",
       "      <td>1106.0</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>8.3014</td>\n",
       "      <td>358500.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-122.24</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1467.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>7.2574</td>\n",
       "      <td>352100.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1274.0</td>\n",
       "      <td>235.0</td>\n",
       "      <td>558.0</td>\n",
       "      <td>219.0</td>\n",
       "      <td>5.6431</td>\n",
       "      <td>341300.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1627.0</td>\n",
       "      <td>280.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>259.0</td>\n",
       "      <td>3.8462</td>\n",
       "      <td>342200.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20635</th>\n",
       "      <td>-121.09</td>\n",
       "      <td>39.48</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1665.0</td>\n",
       "      <td>374.0</td>\n",
       "      <td>845.0</td>\n",
       "      <td>330.0</td>\n",
       "      <td>1.5603</td>\n",
       "      <td>78100.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20636</th>\n",
       "      <td>-121.21</td>\n",
       "      <td>39.49</td>\n",
       "      <td>18.0</td>\n",
       "      <td>697.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>356.0</td>\n",
       "      <td>114.0</td>\n",
       "      <td>2.5568</td>\n",
       "      <td>77100.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20637</th>\n",
       "      <td>-121.22</td>\n",
       "      <td>39.43</td>\n",
       "      <td>17.0</td>\n",
       "      <td>2254.0</td>\n",
       "      <td>485.0</td>\n",
       "      <td>1007.0</td>\n",
       "      <td>433.0</td>\n",
       "      <td>1.7000</td>\n",
       "      <td>92300.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20638</th>\n",
       "      <td>-121.32</td>\n",
       "      <td>39.43</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1860.0</td>\n",
       "      <td>409.0</td>\n",
       "      <td>741.0</td>\n",
       "      <td>349.0</td>\n",
       "      <td>1.8672</td>\n",
       "      <td>84700.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20639</th>\n",
       "      <td>-121.24</td>\n",
       "      <td>39.37</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2785.0</td>\n",
       "      <td>616.0</td>\n",
       "      <td>1387.0</td>\n",
       "      <td>530.0</td>\n",
       "      <td>2.3886</td>\n",
       "      <td>89400.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20640 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0        -122.23     37.88                41.0        880.0           129.0   \n",
       "1        -122.22     37.86                21.0       7099.0          1106.0   \n",
       "2        -122.24     37.85                52.0       1467.0           190.0   \n",
       "3        -122.25     37.85                52.0       1274.0           235.0   \n",
       "4        -122.25     37.85                52.0       1627.0           280.0   \n",
       "...          ...       ...                 ...          ...             ...   \n",
       "20635    -121.09     39.48                25.0       1665.0           374.0   \n",
       "20636    -121.21     39.49                18.0        697.0           150.0   \n",
       "20637    -121.22     39.43                17.0       2254.0           485.0   \n",
       "20638    -121.32     39.43                18.0       1860.0           409.0   \n",
       "20639    -121.24     39.37                16.0       2785.0           616.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "0           322.0       126.0         8.3252            452600.0   \n",
       "1          2401.0      1138.0         8.3014            358500.0   \n",
       "2           496.0       177.0         7.2574            352100.0   \n",
       "3           558.0       219.0         5.6431            341300.0   \n",
       "4           565.0       259.0         3.8462            342200.0   \n",
       "...           ...         ...            ...                 ...   \n",
       "20635       845.0       330.0         1.5603             78100.0   \n",
       "20636       356.0       114.0         2.5568             77100.0   \n",
       "20637      1007.0       433.0         1.7000             92300.0   \n",
       "20638       741.0       349.0         1.8672             84700.0   \n",
       "20639      1387.0       530.0         2.3886             89400.0   \n",
       "\n",
       "      ocean_proximity  \n",
       "0            NEAR BAY  \n",
       "1            NEAR BAY  \n",
       "2            NEAR BAY  \n",
       "3            NEAR BAY  \n",
       "4            NEAR BAY  \n",
       "...               ...  \n",
       "20635          INLAND  \n",
       "20636          INLAND  \n",
       "20637          INLAND  \n",
       "20638          INLAND  \n",
       "20639          INLAND  \n",
       "\n",
       "[20640 rows x 10 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema = pa.DataFrameSchema({\"ocean_proximity\": pa.Column(pa.String)})\n",
    "schema.validate(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-122.23</td>\n",
       "      <td>37.88</td>\n",
       "      <td>41.0</td>\n",
       "      <td>880.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>8.3252</td>\n",
       "      <td>452600.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-122.22</td>\n",
       "      <td>37.86</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7099.0</td>\n",
       "      <td>1106.0</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>8.3014</td>\n",
       "      <td>358500.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-122.24</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1467.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>7.2574</td>\n",
       "      <td>352100.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1274.0</td>\n",
       "      <td>235.0</td>\n",
       "      <td>558.0</td>\n",
       "      <td>219.0</td>\n",
       "      <td>5.6431</td>\n",
       "      <td>341300.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1627.0</td>\n",
       "      <td>280.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>259.0</td>\n",
       "      <td>3.8462</td>\n",
       "      <td>342200.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20635</th>\n",
       "      <td>-121.09</td>\n",
       "      <td>39.48</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1665.0</td>\n",
       "      <td>374.0</td>\n",
       "      <td>845.0</td>\n",
       "      <td>330.0</td>\n",
       "      <td>1.5603</td>\n",
       "      <td>78100.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20636</th>\n",
       "      <td>-121.21</td>\n",
       "      <td>39.49</td>\n",
       "      <td>18.0</td>\n",
       "      <td>697.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>356.0</td>\n",
       "      <td>114.0</td>\n",
       "      <td>2.5568</td>\n",
       "      <td>77100.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20637</th>\n",
       "      <td>-121.22</td>\n",
       "      <td>39.43</td>\n",
       "      <td>17.0</td>\n",
       "      <td>2254.0</td>\n",
       "      <td>485.0</td>\n",
       "      <td>1007.0</td>\n",
       "      <td>433.0</td>\n",
       "      <td>1.7000</td>\n",
       "      <td>92300.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20638</th>\n",
       "      <td>-121.32</td>\n",
       "      <td>39.43</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1860.0</td>\n",
       "      <td>409.0</td>\n",
       "      <td>741.0</td>\n",
       "      <td>349.0</td>\n",
       "      <td>1.8672</td>\n",
       "      <td>84700.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20639</th>\n",
       "      <td>-121.24</td>\n",
       "      <td>39.37</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2785.0</td>\n",
       "      <td>616.0</td>\n",
       "      <td>1387.0</td>\n",
       "      <td>530.0</td>\n",
       "      <td>2.3886</td>\n",
       "      <td>89400.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20640 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0        -122.23     37.88                41.0        880.0           129.0   \n",
       "1        -122.22     37.86                21.0       7099.0          1106.0   \n",
       "2        -122.24     37.85                52.0       1467.0           190.0   \n",
       "3        -122.25     37.85                52.0       1274.0           235.0   \n",
       "4        -122.25     37.85                52.0       1627.0           280.0   \n",
       "...          ...       ...                 ...          ...             ...   \n",
       "20635    -121.09     39.48                25.0       1665.0           374.0   \n",
       "20636    -121.21     39.49                18.0        697.0           150.0   \n",
       "20637    -121.22     39.43                17.0       2254.0           485.0   \n",
       "20638    -121.32     39.43                18.0       1860.0           409.0   \n",
       "20639    -121.24     39.37                16.0       2785.0           616.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "0           322.0       126.0         8.3252            452600.0   \n",
       "1          2401.0      1138.0         8.3014            358500.0   \n",
       "2           496.0       177.0         7.2574            352100.0   \n",
       "3           558.0       219.0         5.6431            341300.0   \n",
       "4           565.0       259.0         3.8462            342200.0   \n",
       "...           ...         ...            ...                 ...   \n",
       "20635       845.0       330.0         1.5603             78100.0   \n",
       "20636       356.0       114.0         2.5568             77100.0   \n",
       "20637      1007.0       433.0         1.7000             92300.0   \n",
       "20638       741.0       349.0         1.8672             84700.0   \n",
       "20639      1387.0       530.0         2.3886             89400.0   \n",
       "\n",
       "      ocean_proximity  \n",
       "0            NEAR BAY  \n",
       "1            NEAR BAY  \n",
       "2            NEAR BAY  \n",
       "3            NEAR BAY  \n",
       "4            NEAR BAY  \n",
       "...               ...  \n",
       "20635          INLAND  \n",
       "20636          INLAND  \n",
       "20637          INLAND  \n",
       "20638          INLAND  \n",
       "20639          INLAND  \n",
       "\n",
       "[20640 rows x 10 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema = pa.DataFrameSchema({\"ocean_proximity\": pa.Column(pa.String,\n",
    "                                                         pa.Check.isin(['NEAR BAY', '<1H OCEAN', 'INLAND', 'NEAR OCEAN', 'ISLAND']))})\n",
    "schema.validate(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['NEAR BAY', '<1H OCEAN', 'INLAND', 'NEAR OCEAN', 'ISLAND'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.ocean_proximity.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-122.23</td>\n",
       "      <td>37.88</td>\n",
       "      <td>41.0</td>\n",
       "      <td>880</td>\n",
       "      <td>129.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>8.3252</td>\n",
       "      <td>452600.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-122.22</td>\n",
       "      <td>37.86</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7099</td>\n",
       "      <td>1106.0</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>8.3014</td>\n",
       "      <td>358500.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-122.24</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1467</td>\n",
       "      <td>190.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>7.2574</td>\n",
       "      <td>352100.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1274</td>\n",
       "      <td>235.0</td>\n",
       "      <td>558.0</td>\n",
       "      <td>219.0</td>\n",
       "      <td>5.6431</td>\n",
       "      <td>341300.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1627</td>\n",
       "      <td>280.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>259.0</td>\n",
       "      <td>3.8462</td>\n",
       "      <td>342200.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20635</th>\n",
       "      <td>-121.09</td>\n",
       "      <td>39.48</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1665</td>\n",
       "      <td>374.0</td>\n",
       "      <td>845.0</td>\n",
       "      <td>330.0</td>\n",
       "      <td>1.5603</td>\n",
       "      <td>78100.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20636</th>\n",
       "      <td>-121.21</td>\n",
       "      <td>39.49</td>\n",
       "      <td>18.0</td>\n",
       "      <td>697</td>\n",
       "      <td>150.0</td>\n",
       "      <td>356.0</td>\n",
       "      <td>114.0</td>\n",
       "      <td>2.5568</td>\n",
       "      <td>77100.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20637</th>\n",
       "      <td>-121.22</td>\n",
       "      <td>39.43</td>\n",
       "      <td>17.0</td>\n",
       "      <td>2254</td>\n",
       "      <td>485.0</td>\n",
       "      <td>1007.0</td>\n",
       "      <td>433.0</td>\n",
       "      <td>1.7000</td>\n",
       "      <td>92300.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20638</th>\n",
       "      <td>-121.32</td>\n",
       "      <td>39.43</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1860</td>\n",
       "      <td>409.0</td>\n",
       "      <td>741.0</td>\n",
       "      <td>349.0</td>\n",
       "      <td>1.8672</td>\n",
       "      <td>84700.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20639</th>\n",
       "      <td>-121.24</td>\n",
       "      <td>39.37</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2785</td>\n",
       "      <td>616.0</td>\n",
       "      <td>1387.0</td>\n",
       "      <td>530.0</td>\n",
       "      <td>2.3886</td>\n",
       "      <td>89400.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20640 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0        -122.23     37.88                41.0          880           129.0   \n",
       "1        -122.22     37.86                21.0         7099          1106.0   \n",
       "2        -122.24     37.85                52.0         1467           190.0   \n",
       "3        -122.25     37.85                52.0         1274           235.0   \n",
       "4        -122.25     37.85                52.0         1627           280.0   \n",
       "...          ...       ...                 ...          ...             ...   \n",
       "20635    -121.09     39.48                25.0         1665           374.0   \n",
       "20636    -121.21     39.49                18.0          697           150.0   \n",
       "20637    -121.22     39.43                17.0         2254           485.0   \n",
       "20638    -121.32     39.43                18.0         1860           409.0   \n",
       "20639    -121.24     39.37                16.0         2785           616.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "0           322.0       126.0         8.3252            452600.0   \n",
       "1          2401.0      1138.0         8.3014            358500.0   \n",
       "2           496.0       177.0         7.2574            352100.0   \n",
       "3           558.0       219.0         5.6431            341300.0   \n",
       "4           565.0       259.0         3.8462            342200.0   \n",
       "...           ...         ...            ...                 ...   \n",
       "20635       845.0       330.0         1.5603             78100.0   \n",
       "20636       356.0       114.0         2.5568             77100.0   \n",
       "20637      1007.0       433.0         1.7000             92300.0   \n",
       "20638       741.0       349.0         1.8672             84700.0   \n",
       "20639      1387.0       530.0         2.3886             89400.0   \n",
       "\n",
       "      ocean_proximity  \n",
       "0            NEAR BAY  \n",
       "1            NEAR BAY  \n",
       "2            NEAR BAY  \n",
       "3            NEAR BAY  \n",
       "4            NEAR BAY  \n",
       "...               ...  \n",
       "20635          INLAND  \n",
       "20636          INLAND  \n",
       "20637          INLAND  \n",
       "20638          INLAND  \n",
       "20639          INLAND  \n",
       "\n",
       "[20640 rows x 10 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "df = pd.read_csv(\"data/housing.csv\", dtype={\"total_rooms\": np.int64})\n",
    "\n",
    "schema = pa.DataFrameSchema({\"ocean_proximity\": pa.Column(pa.String,\n",
    "                                                         pa.Check.isin(['NEAR BAY', '<1H OCEAN', 'INLAND', 'NEAR OCEAN', 'ISLAND'])),\n",
    "                            \"total_rooms\": pa.Column(pa.Int)})\n",
    "schema.validate(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-122.23</td>\n",
       "      <td>37.88</td>\n",
       "      <td>41.0</td>\n",
       "      <td>880</td>\n",
       "      <td>129.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>8.3252</td>\n",
       "      <td>452600.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-122.22</td>\n",
       "      <td>37.86</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7099</td>\n",
       "      <td>1106.0</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>8.3014</td>\n",
       "      <td>358500.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-122.24</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1467</td>\n",
       "      <td>190.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>7.2574</td>\n",
       "      <td>352100.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1274</td>\n",
       "      <td>235.0</td>\n",
       "      <td>558.0</td>\n",
       "      <td>219.0</td>\n",
       "      <td>5.6431</td>\n",
       "      <td>341300.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1627</td>\n",
       "      <td>280.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>259.0</td>\n",
       "      <td>3.8462</td>\n",
       "      <td>342200.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20635</th>\n",
       "      <td>-121.09</td>\n",
       "      <td>39.48</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1665</td>\n",
       "      <td>374.0</td>\n",
       "      <td>845.0</td>\n",
       "      <td>330.0</td>\n",
       "      <td>1.5603</td>\n",
       "      <td>78100.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20636</th>\n",
       "      <td>-121.21</td>\n",
       "      <td>39.49</td>\n",
       "      <td>18.0</td>\n",
       "      <td>697</td>\n",
       "      <td>150.0</td>\n",
       "      <td>356.0</td>\n",
       "      <td>114.0</td>\n",
       "      <td>2.5568</td>\n",
       "      <td>77100.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20637</th>\n",
       "      <td>-121.22</td>\n",
       "      <td>39.43</td>\n",
       "      <td>17.0</td>\n",
       "      <td>2254</td>\n",
       "      <td>485.0</td>\n",
       "      <td>1007.0</td>\n",
       "      <td>433.0</td>\n",
       "      <td>1.7000</td>\n",
       "      <td>92300.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20638</th>\n",
       "      <td>-121.32</td>\n",
       "      <td>39.43</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1860</td>\n",
       "      <td>409.0</td>\n",
       "      <td>741.0</td>\n",
       "      <td>349.0</td>\n",
       "      <td>1.8672</td>\n",
       "      <td>84700.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20639</th>\n",
       "      <td>-121.24</td>\n",
       "      <td>39.37</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2785</td>\n",
       "      <td>616.0</td>\n",
       "      <td>1387.0</td>\n",
       "      <td>530.0</td>\n",
       "      <td>2.3886</td>\n",
       "      <td>89400.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20640 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0        -122.23     37.88                41.0          880           129.0   \n",
       "1        -122.22     37.86                21.0         7099          1106.0   \n",
       "2        -122.24     37.85                52.0         1467           190.0   \n",
       "3        -122.25     37.85                52.0         1274           235.0   \n",
       "4        -122.25     37.85                52.0         1627           280.0   \n",
       "...          ...       ...                 ...          ...             ...   \n",
       "20635    -121.09     39.48                25.0         1665           374.0   \n",
       "20636    -121.21     39.49                18.0          697           150.0   \n",
       "20637    -121.22     39.43                17.0         2254           485.0   \n",
       "20638    -121.32     39.43                18.0         1860           409.0   \n",
       "20639    -121.24     39.37                16.0         2785           616.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "0           322.0       126.0         8.3252            452600.0   \n",
       "1          2401.0      1138.0         8.3014            358500.0   \n",
       "2           496.0       177.0         7.2574            352100.0   \n",
       "3           558.0       219.0         5.6431            341300.0   \n",
       "4           565.0       259.0         3.8462            342200.0   \n",
       "...           ...         ...            ...                 ...   \n",
       "20635       845.0       330.0         1.5603             78100.0   \n",
       "20636       356.0       114.0         2.5568             77100.0   \n",
       "20637      1007.0       433.0         1.7000             92300.0   \n",
       "20638       741.0       349.0         1.8672             84700.0   \n",
       "20639      1387.0       530.0         2.3886             89400.0   \n",
       "\n",
       "      ocean_proximity  \n",
       "0            NEAR BAY  \n",
       "1            NEAR BAY  \n",
       "2            NEAR BAY  \n",
       "3            NEAR BAY  \n",
       "4            NEAR BAY  \n",
       "...               ...  \n",
       "20635          INLAND  \n",
       "20636          INLAND  \n",
       "20637          INLAND  \n",
       "20638          INLAND  \n",
       "20639          INLAND  \n",
       "\n",
       "[20640 rows x 10 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema = pa.DataFrameSchema({\"ocean_proximity\": pa.Column(pa.String,\n",
    "                                                         pa.Check.isin(['NEAR BAY', '<1H OCEAN', 'INLAND', 'NEAR OCEAN', 'ISLAND'])),\n",
    "                            \"total_rooms\": pa.Column(pa.Int),\n",
    "                            \"housing_median_age\": pa.Column(pa.Float, pa.Check(lambda n: n**2 > 0))})\n",
    "schema.validate(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Example Why:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>percentages</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.990</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   percentages\n",
       "0        0.100\n",
       "1        0.300\n",
       "2        0.253\n",
       "3        0.041\n",
       "4        0.210\n",
       "5        0.990"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_simple = pd.DataFrame({\"percentages\": [0.1, 0.3, 25.3, 4.1, 0.21, 99]})\n",
    "df_simple.percentages[df_simple.percentages>1] /= 100\n",
    "schema = pa.DataFrameSchema({\"percentages\": pa.Column(pa.Float,\n",
    "                                                     pa.Check.less_than_or_equal_to(1))})\n",
    "schema.validate(df_simple)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "Explore custom validations and loading data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "schema = pa.DataFrameSchema(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Pandera Schema Validation](https://pandera.readthedocs.io/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}