From 040af7ad74dd43d489a4264471151580d8081159 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Henrik=20Bostr=C3=B6m?= Date: Wed, 28 Jun 2023 21:46:22 +0200 Subject: [PATCH] 0.6.0 --- CHANGELOG.md | 14 +++ README.md | 288 ++++++++++++++++++++++++++++++++++++++++++++------- setup.py | 4 +- 3 files changed, 268 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 293f812..c05a3c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,19 @@ # Changelog +## v0.6.0 (28/06/2023) + +### Features + +- The classes `ConformalClassifier` and `WrapClassifier` have been added to `crepes`, allowing for generation of standard and Mondrian conformal classifiers, which produce p-values and prediction sets. The `calibrate` method of `WrapClassifier` allows for easily generating class-conditional conformal classifiers and using out-of-bag calibration. See [the documentation](https://crepes.readthedocs.io/en/latest/crepes.html) for the interface to objects of the class through the `calibrate`, `predict_p` and `predict_set` methods, in addition to the `fit`, `predict` and `predict_proba` methods of the wrapped learner. The method `evaluate` allows for evaluating the predictive performance using a set of standard metrics. + +- The function `hinge` for computing non-conformity scores for conformal classifiers has been added to `crepes.extras`. + +### Fixes + +- The class `Wrap` has changed name to `WrapRegressor` and the arguments to the `calibrate` method of this class have been changed to be in line with the `calibrate` method of `WrapClassifier`. + +- The Jupyter notebooks `crepes_nb_wrap.ipynb` and `crepes_nb.ipynb` have been updated and extended + ## v0.5.1 (22/06/2023) ### Fix diff --git a/README.md b/README.md index 73a6617..019e88c 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -

crepes

+

crepes

PyPI version @@ -11,9 +11,19 @@
-`crepes` is a Python package for generating *conformal regressors*, which transform point predictions of any underlying regression model into prediction intervals for specified levels of confidence. The package also implements *conformal predictive systems*, which transform the point predictions into cumulative distribution functions. +`crepes` is a Python package that implements conformal classifiers, +regressors, and predictive systems, on top of any standard classifier +and regressor, transforming the original predictions into +well-calibrated p-values and cumulative distribution functions, or +prediction sets and intervals with coverage guarantees. -The `crepes` package implements standard, normalized and Mondrian conformal regressors and predictive systems. While the package allows you to use your own difficulty estimates and Mondrian categories, there is also a separate module, called `crepes.extras`, which provides some standard options for these. +The `crepes` package implements standard and Mondrian conformal +classifiers as well as standard, normalized and Mondrian conformal +regressors and predictive systems. While the package allows you to use +your own functions to compute difficulty estimates, non-conformity +scores and Mondrian categories, there is also a separate module, +called `crepes.extras`, which provides some standard options for +these. ## Installation @@ -35,15 +45,169 @@ For the complete documentation, see [crepes.readthedocs.io](https://crepes.readt ## Quickstart -Let us illustrate the use of `crepes` by importing a dataset from [www.openml.org](https://www.openml.org), -which we split into a training and a test set using `train_test_split` from [sklearn](https://scikit-learn.org), -and then further split the training set into a proper training set and a calibration set: +Let us illustrate how we may use `crepes` to generate and apply +conformal classifiers with a dataset from +[www.openml.org](https://www.openml.org), which we first split into a +training and a test set using `train_test_split` from +[sklearn](https://scikit-learn.org), and then further split the +training set into a proper training set and a calibration set: + +```python +from crepes import WrapClassifier +from sklearn.ensemble import RandomForestClassifier + +dataset = fetch_openml(name="qsar-biodeg", parser="auto") + +X = dataset.data.values.astype(float) +y = dataset.target.values + +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5) + +X_prop_train, X_cal, y_prop_train, y_cal = train_test_split(X_train, y_train, + test_size=0.25) +``` + +We now "wrap" a random forest classifier, fit it to the proper +training set, and fit a standard conformal classifier through the +`calibrate` method: + +```python +rf = WrapClassifier(RandomForestClassifier(n_jobs=-1)) + +rf.fit(X_prop_train, y_prop_train) + +rf.calibrate(X_cal, y_cal) +``` + +We may now produce p-values for the test set (an array with as many +columns as there are classes): + +```python +rf.predict_p(X_test) +``` + +```numpy +array([[0.46552707, 0.04407598], + [0.00382577, 0.85400826], + [0.64930738, 0.00804963], + ..., + [0.33376105, 0.04065675], + [0.16968437, 0.12810237], + [0.02346899, 0.49634959]]) +``` + +We can also get prediction sets, represented by binary vectors +indicating presence (1) or absence (0) of the class labels that +correspond to the columns, here at the 90% confidence level: + +```python +rf.predict_set(X_test, confidence=0.9) +``` + +```numpy +array([[1, 0], + [0, 1], + [1, 0], + ..., + [1, 0], + [1, 1], + [0, 1]]) +``` + +Since we have access to the true class labels, we can evaluate the +conformal classifier (here using all available metrics which is the +default): + +```python +rf.evaluate(X_test, y_test, confidence=0.9) +``` + +```python +{'error': 0.11553030303030298, + 'avg_c': 1.0776515151515151, + 'one_c': 0.9223484848484849, + 'empty': 0.0, + 'time_fit': 2.7418136596679688e-05, + 'time_evaluate': 0.01745915412902832} +``` + +To control the error level across different groups of objects of +interest, we may use so-called Mondrian conformal classifiers. A +Mondrian conformal classifier is formed by providing the names of the +categories as an additional argument, named `bins`, for the +`calibrate` method. + +Here we consider two categories formed by whether the third column (number of heavy atoms) equals zero or not: + +```python +bins_cal = X_cal[:,2] == 0 + +rf_mond = WrapClassifier(rf.learner) + +rf_mond.calibrate(X_cal, y_cal, bins=bins_cal) + +bins_test = X_test[:,2] == 0 + +rf_mond.predict_set(X_test, bins=bins_test) +``` + +```numpy +array([[1, 0], + [0, 1], + [1, 0], + ..., + [1, 1], + [1, 1], + [0, 1]]) +``` + +For conformal classifiers that employ learners that use bagging, like +random forests, we may consider an alternative strategy to dividing +the original training set into a proper training and calibration set; +we may use the out-of-bag (OOB) predictions, which allow us to use the +full training set for both model building and calibration. It should +be noted that this strategy does not come with the theoretical +validity guarantee of the above (inductive) conformal classifiers, due +to that calibration and test instances are not handled in exactly the +same way. In practice, however, conformal classifiers based on +out-of-bag predictions rarely fail to meet the coverage requirements. + +Below we show how to enable this in conjunction with a specific type +of Mondrian conformal classifier, a so-called class-conditional +conformal classifier, which uses the class labels as Mondrian +categories: + +```python +rf = WrapClassifier(RandomForestClassifier(n_jobs=-1, n_estimators=500, oob_score=True)) + +rf.fit(X_train, y_train) + +rf.calibrate(X_train, y_train, class_cond=True, oob=True) + +rf.evaluate(X_test, y_test, confidence=0.99) +``` + +```python +{'error': 0.009469696969697017, + 'avg_c': 1.696969696969697, + 'one_c': 0.30303030303030304, + 'empty': 0.0, + 'time_fit': 0.0002560615539550781, + 'time_evaluate': 0.06656742095947266} +``` + +Let us also illustrate how `crepes` can be used to generate conformal +regressors and predictive systems. Again, we import a dataset from +[www.openml.org](https://www.openml.org), which we split into a +training and a test set and then further split the training set into a +proper training set and a calibration set: ```python from sklearn.datasets import fetch_openml from sklearn.model_selection import train_test_split -dataset = fetch_openml(name="house_sales", version=3) +dataset = fetch_openml(name="house_sales", version=3, parser="auto") + X = dataset.data.values.astype(float) y = dataset.target.values.astype(float) @@ -52,26 +216,28 @@ X_prop_train, X_cal, y_prop_train, y_cal = train_test_split(X_train, y_train, test_size=0.25) ``` -Let us now "wrap" a `RandomForestRegressor` from [sklearn](https://scikit-learn.org) using the class `Wrap` from `crepes` -and fit it (in the usual way) to the proper training set: +Let us now "wrap" a `RandomForestRegressor` from +[sklearn](https://scikit-learn.org) using the class `WrapRegressor` +from `crepes` and fit it (in the usual way) to the proper training +set: ```python from sklearn.ensemble import RandomForestRegressor -from crepes import Wrap +from crepes import WrapRegressor -rf = Wrap(RandomForestRegressor()) +rf = WrapRegressor(RandomForestRegressor()) rf.fit(X_prop_train, y_prop_train) ``` -We can use the fitted model to obtain point predictions (again, in the usual way) for the calibration objects, from which we can calculate the residuals. -These residuals are exactly what we need to "calibrate" the learner: +We may now fit a conformal regressor using the calibration set through +the `calibrate` method: ```python -residuals = y_cal - rf.predict(X_cal) -rf.calibrate(residuals) +rf.calibrate(X_cal, y_cal) ``` -A (standard) conformal regressor was formed (under the hood). We may now use it for obtaining prediction intervals for the test set, here using a confidence level of 99%: +The conformal regressor can now produce prediction intervals for the +test set, here using a confidence level of 99%: ```python rf.predict_int(X_test, confidence=0.99) @@ -87,9 +253,13 @@ array([[-171902.2 , 953866.2 ], [-227057.4 , 898711. ]]) ``` -The output is a [NumPy](https://numpy.org) array with a row for each test instance, and where the two columns specify the lower and upper bound of each prediction interval. +The output is a [NumPy](https://numpy.org) array with a row for each +test instance, and where the two columns specify the lower and upper +bound of each prediction interval. -We may request that the intervals are cut to exclude impossible values, in this case below 0, and if we also rely on the default confidence level (0.95), the output intervals will be a bit tighter: +We may request that the intervals are cut to exclude impossible +values, in this case below 0, and if we also rely on the default +confidence level (0.95), the output intervals will be a bit tighter: ```python rf.predict_int(X_test, y_min=0) @@ -105,9 +275,18 @@ array([[ 152258.55, 629705.45], [ 97103.35, 574550.25]]) ``` -The above intervals are not normalized, i.e., they are all of the same size (at least before they are cut). We could make them more informative through normalization using difficulty estimates; objects considered more difficult will be assigned wider intervals. +The above intervals are not normalized, i.e., they are all of the same +size (at least before they are cut). We could make them more +informative through normalization using difficulty estimates; objects +considered more difficult will be assigned wider intervals. -We will use a `DifficultyEstimator` from the `crepes.extras` module for this purpose. Here we estimate the difficulty by the standard deviation of the target of the k (default `k=25`) nearest neighbors in the proper training set to each object in the calibration set. A small value (beta) is added to the estimates, which may be given through an argument to the function; below we just use the default, i.e., `beta=0.01`. +We will use a `DifficultyEstimator` from the `crepes.extras` module +for this purpose. Here we estimate the difficulty by the standard +deviation of the target of the k (default `k=25`) nearest neighbors in +the proper training set to each object in the calibration set. A small +value (beta) is added to the estimates, which may be given through an +argument to the function; below we just use the default, i.e., +`beta=0.01`. We first obtain the difficulty estimates for the calibration set: @@ -120,13 +299,16 @@ de.fit(X_prop_train, y=y_prop_train) sigmas_cal = de.apply(X_cal) ``` -These can now be used for the calibration, which (under the hood) will produce a normalized conformal regressor: +These can now be used for the calibration, which will produce a +normalized conformal regressor: ```python -rf.calibrate(residuals, sigmas=sigmas_cal) +rf.calibrate(X_cal, y_cal, sigmas=sigmas_cal) ``` -We need difficulty estimates for the test set too, which we provide as input to `predict_int`: +We need difficulty estimates for the test set too, which we provide as +input to `predict_int`: + ```python sigmas_test = de.apply(X_test) rf.predict_int(X_test, sigmas=sigmas_test, y_min=0) @@ -142,11 +324,27 @@ array([[ 226719.06607977, 555244.93392023], [ 145340.39076824, 526313.20923176]]) ``` -Depending on the employed difficulty estimator, the normalized intervals may sometimes be unreasonably large, in the sense that they may be several times larger than any previously observed error. Moreover, if the difficulty estimator is uninformative, e.g., completely random, the varying interval sizes may give a false impression of that we can expect lower prediction errors for instances with tighter intervals. Ideally, a difficulty estimator providing little or no information on the expected error should instead lead to more uniformly distributed interval sizes. - -A Mondrian conformal regressor can be used to address these problems, by dividing the object space into non-overlapping so-called Mondrian categories, and forming a (standard) conformal regressor for each category. The category membership of the objects can be provided as an additional argument, named `bins`, for the `fit` method. - -Here we use the helper function `binning` from `crepes.extras` to form Mondrian categories by equal-sized binning of the difficulty estimates; the function returns labels for the calibration objects the we provide as input to the calibration, and we also get thresholds for the bins, which can use later when binning the test objects: +Depending on the employed difficulty estimator, the normalized +intervals may sometimes be unreasonably large, in the sense that they +may be several times larger than any previously observed +error. Moreover, if the difficulty estimator is uninformative, e.g., +completely random, the varying interval sizes may give a false +impression of that we can expect lower prediction errors for instances +with tighter intervals. Ideally, a difficulty estimator providing +little or no information on the expected error should instead lead to +more uniformly distributed interval sizes. + +A Mondrian conformal regressor can be used to address these problems, +by dividing the object space into non-overlapping so-called Mondrian +categories, and forming a (standard) conformal regressor for each +category. The category membership of the objects can be provided as an +additional argument, named `bins`, for the `fit` method. + +Here we use the helper function `binning` from `crepes.extras` to form +Mondrian categories by equal-sized binning of the difficulty +estimates; the function returns labels for the calibration objects the +we provide as input to the calibration, and we also get thresholds for +the bins, which can use later when binning the test objects: ```python from crepes.extras import binning @@ -155,7 +353,8 @@ bins_cal, bin_thresholds = binning(sigmas_cal, bins=20) rf.calibrate(residuals, bins=bins_cal) ``` -Let us now get the labels of the Mondrian categories for the test objects and use them when predicting intervals: +Let us now get the labels of the Mondrian categories for the test +objects and use them when predicting intervals: ```python bins_test = binning(sigmas_test, bins=bin_thresholds) @@ -172,15 +371,24 @@ array([[ 206379.7 , 575584.3 ], [ 140587.46, 531066.14]]) ``` -We could very easily switch from conformal regressors to conformal predictive systems. The latter produce cumulative distribution functions (conformal predictive distributions). From these we can generate prediction intervals, but we can also obtain percentiles, calibrated point predictions, as well as p-values for given target values. Let us see how we can go ahead to do that. +We could very easily switch from conformal regressors to conformal +predictive systems. The latter produce cumulative distribution +functions (conformal predictive distributions). From these we can +generate prediction intervals, but we can also obtain percentiles, +calibrated point predictions, as well as p-values for given target +values. Let us see how we can go ahead to do that. -Well, there is only one thing above that changes: just provide `cps=True` to the `calibrate` method. +Well, there is only one thing above that changes: just provide +`cps=True` to the `calibrate` method. -We can, for example, form normalized Mondrian conformal predictive systems, by providing both `bins` and `sigmas` to the `calibrate` method. Here we will consider Mondrian categories formed from binning the point predictions: +We can, for example, form normalized Mondrian conformal predictive +systems, by providing both `bins` and `sigmas` to the `calibrate` +method. Here we will consider Mondrian categories formed from binning +the point predictions: ```python bins_cal, bin_thresholds = binning(rf.predict(X_cal), bins=5) -rf.calibrate(residuals, sigmas=sigmas_cal, bins=bins_cal, cps=True) +rf.calibrate(X_cal, y_cal, sigmas=sigmas_cal, bins=bins_cal, cps=True) ``` By providing the bins (and sigmas) for the test objects, we can now make predictions with the conformal predictive system, through the method `predict_cps`. @@ -213,10 +421,18 @@ array([0.98603614, 0.87178256, 0.44201984, ..., 0.05688804, 0.09473604, 0.31069913]) ``` -We may request that the `predict_cps` method returns the full conformal predictive distribution (CPD) for each test instance, as defined by the threshold values, by setting `return_cpds=True`. The format of the distributions vary with the type of conformal predictive system; for a standard and normalized CPS, the output is an array with a row for each test instance and a column for each calibration instance (residual), while for a Mondrian CPS, the default output is a vector containing one CPD per test instance, since the number of values may vary between categories. +We may request that the `predict_cps` method returns the full +conformal predictive distribution (CPD) for each test instance, as +defined by the threshold values, by setting `return_cpds=True`. The +format of the distributions vary with the type of conformal predictive +system; for a standard and normalized CPS, the output is an array with +a row for each test instance and a column for each calibration +instance (residual), while for a Mondrian CPS, the default output is a +vector containing one CPD per test instance, since the number of +values may vary between categories. ```python -rf.predict_cps(X_test, sigmas=sigmas_test, bins=bins_test, return_cpds=True) +cpds = rf.predict_cps(X_test, sigmas=sigmas_test, bins=bins_test, return_cpds=True) ``` The resulting vector of arrays is not displayed here, but we instead provide a plot for the CPD of a random test instance: @@ -225,7 +441,7 @@ The resulting vector of arrays is not displayed here, but we instead provide a p ## Examples -For additional examples of how to use the package and module, including how to use out-of-bag predictions rather than having to rely on dividing the training set into a proper training and calibration set, see [the documentation](https://crepes.readthedocs.io/en/latest/), [this Jupyter notebook using Wrap](https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb), and [this Jupyter notebook using ConformalRegressor and ConformalPredictiveSystem](https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb). +For additional examples of how to use the package and module, see [the documentation](https://crepes.readthedocs.io/en/latest/), [this Jupyter notebook using WrapClassifier and WrapRegressor](https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb_wrap.ipynb), and [this Jupyter notebook using ConformalClassifier, ConformalRegressor, and ConformalPredictiveSystem](https://github.com/henrikbostrom/crepes/blob/main/docs/crepes_nb.ipynb). ## Citing crepes diff --git a/setup.py b/setup.py index 23f1d46..74cd762 100644 --- a/setup.py +++ b/setup.py @@ -5,10 +5,10 @@ setuptools.setup( name="crepes", - version="0.5.1", + version="0.6.0", author="Henrik Boström", author_email="bostromh@kth.se", - description="Conformal regressors and predictive systems (crepes)", + description="Conformal classifiers, regressors, and predictive systems (crepes)", long_description=long_description, long_description_content_type="text/markdown", url="https://github.com/henrikbostrom/crepes",