notebooks/2_training_pipeline_data_validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Data Validation in Training Pipelines\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In this notebook, we will go through the process of validating dataframes in a training pipeline using Pandera."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NatanMish/data_validation/blob/main/notebooks/2_training_pipeline_data_validation.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Install the required packages and import them to the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install sklearn pandera\\[strategies\\]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import pandera as pa\n",
    "from typing import Optional"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "house_data = pd.read_csv('https://github.com/NatanMish/data_validation/blob/a77b247b25c6622ce0c8f8cbc505228161c31a3c/data/train.csv?raw=true')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Train basic model\n",
    "We'll start by setting up a training pipeline using Scikit Learn's native class. We only want to select a few basic features for the purpose of this example, so we'll set up a pipeline step class that will select only those features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class ChooseFeatures(BaseEstimator):\n",
    "    def __init__(self, features=None):\n",
    "        self.features = features\n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    def transform(self, X, y=None):\n",
    "        return X[self.features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "feature_names = ['LotArea','YearBuilt','1stFlrSF','2ndFlrSF','FullBath','BedroomAbvGr','TotRmsAbvGrd', 'LotFrontage']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we set up the pipeline and fit it to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pipe = Pipeline([\n",
    "     ('feature_selection', ChooseFeatures(features=feature_names)),\n",
    "     ('scaler', StandardScaler()),\n",
    "     ('rf', RandomForestRegressor())\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "X = house_data\n",
    "y = house_data.SalePrice\n",
    "pipe.fit(house_data, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Looks like our data has null values and this causes the model to break. Let's take a look at Pandera to see how it can help us with this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"https://raw.githubusercontent.com/pandera-dev/pandera/master/docs/source/_static/pandera-banner.png\" width=\"500\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Pandera provides a flexible and expressive API for performing data validation on dataframes to make data processing pipelines more readable and robust. Dataframes contain information that pandera explicitly validates at runtime. This is useful in production-critical data pipelines or reproducible research settings. We'll take a look at these Pandera features:\n",
    "\n",
    "1. Check the types and properties of columns in a pd.DataFrame or values in a pd.Series.\n",
    "\n",
    "2. Perform more complex statistical validation like hypothesis testing.\n",
    "\n",
    "3. Integrate with existing data analysis/processing pipelines via function decorators.\n",
    "\n",
    "4. Define schema models with the class-based API with pydantic-style syntax and validate dataframes using the typing syntax.\n",
    "\n",
    "5. Synthesize data from schema objects for property-based testing with pandas data structures.\n",
    "\n",
    "6. Lazily Validate dataframes so that all validation rules are executed before raising an error.\n",
    "\n",
    "For more information, see [Pandera's documentation](https://pandera.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 1. DataFrame Schemas - Type Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# We'll add one more feature to make it more interesting\n",
    "feature_names.append('LotConfig')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Create a basic schema for the house_data DataFrame to check types for just 2 of the feature\n",
    "basic_types_schema = pa.DataFrameSchema({\n",
    "    \"LotArea\": pa.Column(int),\n",
    "    \"LotConfig\": pa.Column(str),\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Validate the house_data DataFrame against the basic_schema\n",
    "# notice that although we only defined two of the features in the dataframe, and Pandera ignored the rest.\n",
    "basic_types_schema.validate(house_data[feature_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "There is an output from the validation, this means that the data is valid.\n",
    "There are different ways we can specify the type:\n",
    "- a string alias, as long as it is recognized by pandas.\n",
    "- a python type: int, float, double, bool, str\n",
    "- a numpy data type\n",
    "- a pandas extension type: it can be an instance (e.g pd.CategoricalDtype([“a”, “b”])) or a class (e.g pandas.CategoricalDtype) if it can be initialized with default values.\n",
    "- a pandera DataType: it can also be an instance or a class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Now let's create a schema that does not fit the data types in house data\n",
    "bad_types_schema = pa.DataFrameSchema({\n",
    "    \"LotArea\": pa.Column(int),\n",
    "    \"LotConfig\": pa.Column(float),\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# The bad schema validation will throw an error\n",
    "bad_types_schema.validate(house_data[feature_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 2. DataFrame Schemas - Value Ranges Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Pandera also allows validating value ranges for numerical columns\n",
    "value_range_schema = pa.DataFrameSchema({\n",
    "    \"LotArea\": pa.Column(int, pa.Check(lambda s: s <= 1000000), nullable=False),\n",
    "    \"YearBuilt\": pa.Column(int, pa.Check.in_range(1800, 2022)),\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Validate the house_data DataFrame against the value_range_schema\n",
    "value_range_schema.validate(house_data[feature_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 3. DataFrame Schemas - Catch Bad Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "What if instead of breaking on error we want to continue processing the dataframe? or we want to skip the bad data? we can use the `failure_cases` attribute of the error message to capture the bad data indices and the `lazy` argument for going over the entire dataframe instead of failing on the first bad row. We can do that by utilizing a try-except block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# We'll use a small sample of the data to make the example more clear\n",
    "sample_data = house_data.sample(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Create a schema that will fail on the first bad data point\n",
    "catch_bad_data_schema = pa.DataFrameSchema({\n",
    "    \"LotArea\": pa.Column(int, pa.Check(lambda s: s <= 1000000)),\n",
    "    \"YearBuilt\": pa.Column(int, pa.Check.in_range(1900,1990)),  # notice that the year built has a restrictive range\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Validating the house_data DataFrame against the catch_bad_data_schema will throw an error\n",
    "catch_bad_data_schema.validate(sample_data[feature_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now let's use a try except block to catch the bad data indices. This is a common and valid practice in Python called EAFP - \"easier to ask for forgiveness than permission\" which might not be as well recieved in other languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    catch_bad_data_schema.validate(sample_data[feature_names], lazy=True)\n",
    "except pa.errors.SchemaErrors as e:\n",
    "    failure_cases = e.failure_cases\n",
    "\n",
    "# Failure cases is a dataframe of the bad data only\n",
    "failure_cases.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# We can easily filter out the bad data from the original dataframe using the failure_cases dataframe\n",
    "filtered_df = sample_data[~sample_data.index.isin(failure_cases[\"index\"])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Let's see that the filtered data passes the validation test\n",
    "catch_bad_data_schema.validate(filtered_df[feature_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 4. DataFrame Schemas - Validate acceptable categorical values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "lot_config_values = [\"Inside\", \"Corner\", \"CulDSac\", \"FR3\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "lot_config_values_schema = pa.DataFrameSchema({\n",
    "    \"LotArea\": pa.Column(int, pa.Check(lambda s: s <= 1000000)),\n",
    "    \"LotConfig\": pa.Column(str, pa.Check.isin(lot_config_values)),\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Validating the house_data DataFrame against the lot_config_values_schema will throw an error\n",
    "lot_config_values_schema.validate(house_data[feature_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Other useful methods for `pa.Check` are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.eq.html\">pandera.checks.Check.eq</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.equal_to.html\">pandera.checks.Check.equal_to</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.ge.html\">pandera.checks.Check.ge</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.greater_than.html\">pandera.checks.Check.greater_than</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.greater_than_or_equal_to.html\">pandera.checks.Check.greater_than_or_equal_to</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.gt.html\">pandera.checks.Check.gt</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.in_range.html\">pandera.checks.Check.in_range</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.isin.html\">pandera.checks.Check.isin</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.le.html\">pandera.checks.Check.le</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.less_than.html\">pandera.checks.Check.less_than</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.less_than_or_equal_to.html\">pandera.checks.Check.less_than_or_equal_to</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.lt.html\">pandera.checks.Check.lt</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.ne.html\">pandera.checks.Check.ne</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.not_equal_to.html\">pandera.checks.Check.not_equal_to</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.notin.html\">pandera.checks.Check.notin</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.str_contains.html\">pandera.checks.Check.str_contains</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.str_endswith.html\">pandera.checks.Check.str_endswith</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.str_length.html\">pandera.checks.Check.str_length</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.str_matches.html\">pandera.checks.Check.str_matches</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.str_startswith.html\">pandera.checks.Check.str_startswith</a></li>\n",
    "<li class=\"toctree-l4\"><a class=\"reference internal\" href=\"methods/pandera.checks.Check.__call__.html\">pandera.checks.Check.__call__</a></li>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 5. DataFrame Schemas - `Coerce`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "`Coerce` allows forcing type onto a specific dataframe column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "house_data.LotArea.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "coerce_schema = pa.DataFrameSchema(\n",
    "    columns={\"LotArea\": pa.Column(float)},\n",
    "    coerce=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "coerce_schema.validate(house_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# and if we set coerce to True, we can coerce the dataframe to the schema\n",
    "coerce_schema = pa.DataFrameSchema(\n",
    "    columns={\"LotArea\": pa.Column(float)},\n",
    "    coerce=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "coerce_schema.validate(house_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 6. DataFrame Schemas - `Strict`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Using `strict` we can specify that the dataframe must have the exact columns specified in the schema\n",
    "strict_schema = pa.DataFrameSchema(\n",
    "    columns={\"LotArea\": pa.Column(int), \"YearBuilt\": pa.Column(int)},\n",
    "    strict=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Another useful feature is setting `strict` to 'filter' which will filter out any columns that are not in the schema\n",
    "strict_filter_schema = pa.DataFrameSchema(\n",
    "    columns={\"LotArea\": pa.Column(int), \"YearBuilt\": pa.Column(int)},\n",
    "    strict=\"filter\",\n",
    ")\n",
    "filtered_df = strict_filter_schema.validate(house_data)\n",
    "filtered_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Exercise 1 - DataFrame Schemas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Create a pa.DataFrameSchema object for the `house_data` DataFrame. Not all the checks requested were shown above, for some of them you'll need to have a quick search in the Pandera documentation. It should have the following columns and rules:\n",
    "1. Id is a required and unique column of an integer type and cannot be null.\n",
    "2. MSZoning is a non-required column of a string type and can be null. If not null it can only accept these values - 'RL', 'RM', 'C (all)', 'RH' and 'FV'.\n",
    "3. OverallQual is a required column of an integer type, cannot be null and must be in the range 1-10.\n",
    "4. BsmtCond is a non-required column of a string type and can be null. If not null it can only accept a string of a length of 2.\n",
    "\n",
    "Bonus:\n",
    "\n",
    "5. Add the 1stFlrSF and 2ndFlrSF columns to the schema and validate that on average 1stFlrSF>=2ndFlrSF.\n",
    "\n",
    "Create the schema such that it filters out any other columns that are not in the schema.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "exercise_schema = pa.DataFrameSchema(\n",
    "    columns={\n",
    "        \"Id\": <YOUR ANSWER HERE>,\n",
    "        \"MSZoning\": <YOUR ANSWER HERE>,\n",
    "        \"OverallQual\": <YOUR ANSWER HERE>,\n",
    "        \"BsmtCond\": <YOUR ANSWER HERE>,\n",
    "        \"1stFlrSF\": <YOUR ANSWER HERE>,\n",
    "        \"2ndFlrSF\": <YOUR ANSWER HERE>,\n",
    "    },\n",
    "    strict=<YOUR ANSWER HERE>,\n",
    "    checks=<YOUR ANSWER HERE>,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "exercise_schema.validate(house_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "*Exercise solutions can be found in the exercise solutions file in the current directory.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 7. Pandera Decorators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Pandera offers decorators which allow a seamless integration of Pandera validations with our code. The available decorators are:\n",
    "- @check_input\n",
    "- @check_output\n",
    "- @check_io\n",
    "- @check_types\n",
    "\n",
    "We will use a different way for defining the schemas in the next example, but the same principles apply. Here we will construct a class based Pandera model which we can use to validate inputs and outputs to our data in Pydantic style syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names.remove('LotConfig')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pandera.typing import Series\n",
    "\n",
    "# Define a class based Pandera model for the input data to the feature engineering step\n",
    "class FeaturesSchemaPreEngineering(pa.SchemaModel):\n",
    "    LotArea: Series[int] = pa.Field(nullable=False, ge=0)\n",
    "    YearBuilt: Series[int] = pa.Field(nullable=False, ge=1700)\n",
    "    FirstFlrSF: Series[int] = pa.Field(nullable=False, ge=0, alias=\"1stFlrSF\") # alias is used to give the column a different name in the schema because the column name starts with a number\n",
    "    SecondFlrSF: Series[int] = pa.Field(nullable=False, ge=0, alias=\"2ndFlrSF\")\n",
    "    FullBath: Series[int] = pa.Field(nullable=False, ge=0)\n",
    "    BedroomAbvGr: Series[int] = pa.Field(nullable=False, ge=0)\n",
    "    TotRmsAbvGrd: Series[int] = pa.Field(nullable=False, ge=0)\n",
    "    LotFrontage: Series[float] = pa.Field(nullable=True, ge=0)\n",
    "    class Config:\n",
    "        strict=True\n",
    "\n",
    "# Define a class based Pandera model for the output data to the feature engineering step, notice how we inherit the FeaturesSchemaPreEngineering class and extend it with the output data schema.\n",
    "class FeaturesSchemaPostEngineering(FeaturesSchemaPreEngineering):\n",
    "    LotFrontage: Series[float] = pa.Field(nullable=False, ge=0)\n",
    "    HouseAge: Series[int] = pa.Field(nullable=False, ge=0)\n",
    "    AllFloorsSF: Series[int] = pa.Field(nullable=False, ge=0)\n",
    "    NonBedRmAbvGrd: Series[int] = pa.Field(nullable=False, ge=0)\n",
    "\n",
    "    class Config:\n",
    "        strict=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pandera import check_types\n",
    "from pandera.typing import DataFrame as DataFramePa\n",
    "@check_types\n",
    "def feature_engineering(df: DataFramePa[FeaturesSchemaPreEngineering]) -> DataFramePa[FeaturesSchemaPostEngineering]:\n",
    "    df = df.copy()\n",
    "    df[\"HouseAge\"] = 2022 - df[\"YearBuilt\"]\n",
    "    df[\"AllFloorsSF\"] = df[\"1stFlrSF\"] + df[\"2ndFlrSF\"]\n",
    "    df[\"NonBedRmAbvGrd\"] = df[\"TotRmsAbvGrd\"] - df[\"BedroomAbvGr\"]\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This run should fail because we haven't filtered out the null values\n",
    "feature_engineering(house_data.loc[:, feature_names])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# This run should complete without error\n",
    "feature_engineering(house_data.loc[~house_data.LotFrontage.isnull(), feature_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 8. Data Synthesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Pandera offers a simple way to generate synthetic data. We can use the `example` method to generate a DataFrame with a given schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "FeaturesSchemaPreEngineering.example(size=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Notice how some of the columns have 'crazy' values, this is because the random data generating process is using the checks we have defined in the schema for detecting the acceptable ranges possible.\n",
    "\n",
    "We can use the hypothesis library to generate data for our schema and then use it in a unit test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import hypothesis\n",
    "@hypothesis.given(FeaturesSchemaPreEngineering.strategy(size=1))\n",
    "def test_processing_fn(dataframe):\n",
    "    feature_engineering(dataframe)\n",
    "    \n",
    "test_processing_fn()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 9. Schema Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Pandera can infer schemas from data. This is useful when you have a large dataset and you don't want to define a schema manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "schema = pa.infer_schema(house_data)\n",
    "print(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Exercise 2 - Incorporating validation in a training pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Let's use what we have learned to implement data validation in a training pipeline.\n",
    "Choose which features to use for training. You can use these, or add your own:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "feature_names = ['LotArea','YearBuilt','1stFlrSF','2ndFlrSF','FullBath','BedroomAbvGr','TotRmsAbvGrd', 'LotFrontage']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "If you chose other features, make sure to include the ones that are required for the schema to be valid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# class FeaturesSchemaPreEngineering(pa.SchemaModel):\n",
    "#     ...\n",
    "\n",
    "# class FeaturesSchemaPostEngineering(FeaturesSchemaPreEngineering):\n",
    "#     ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "@check_types\n",
    "def feat_eng_step_1(df: DataFramePa[FeaturesSchemaPreEngineering]) -> DataFramePa[FeaturesSchemaPostEngineering]:\n",
    "    df = df.copy()\n",
    "    df[\"HouseAge\"] = 2022 - df[\"YearBuilt\"]\n",
    "    df[\"AllFloorsSF\"] = df[\"1stFlrSF\"] + df[\"2ndFlrSF\"]\n",
    "    df[\"NonBedRmAbvGrd\"] = df[\"TotRmsAbvGrd\"] - df[\"BedroomAbvGr\"]\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Fill in the missing steps below:\n",
    "1. First try to run the feat_eng_step_1 function on the X dataframe, and return a tuple of the feature engineered dataframe and the y series.\n",
    "2. If a pa.errors.SchemaError is raised, capture the exception and extract the `failure_cases` property from the exception. Filter out the invalid data indices from X and y, and then return a tuple comprised of:\n",
    "    - the filtered X dataframe\n",
    "    - the filtered y series\n",
    "    - the failure cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def feat_eng_all_steps(X: pd.DataFrame, y: pd.Series) -> (pd.DataFrame, pd.Series, Optional[pd.DataFrame]):\n",
    "    try:\n",
    "        <YOUR CODE HERE>\n",
    "    except <YOUR CODE HERE> as e:\n",
    "        <YOUR CODE HERE>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Same pipeline as defined previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pipe = Pipeline([\n",
    "    ('feature_selection', ChooseFeatures(features=feature_names)),\n",
    "    ('scaler', StandardScaler()),\n",
    "    ('rf', RandomForestRegressor())\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Running the cell below should fit the data to the pipeline, this time with no errors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "X, y, isolated_invalid_data  = feat_eng_all_steps(house_data[feature_names], house_data[\"SalePrice\"])\n",
    "pipe.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# And we can see the invalid data we were unable to fit:\n",
    "isolated_invalid_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}