allow grid search for classifiers/regressors params in ensemble methods (#259)

Author: rasbt

docs/sources/CHANGELOG.md

@@ -18,6 +18,7 @@ The CHANGELOG for the current development version is available at
 - Added `evaluate.permutation_test`, a permutation test for hypothesis testing (or A/B testing) to test if two samples come from the same distribution. Or in other words, a procedure to test the null hypothesis that that two groups are not significantly different (e.g., a treatment and a control group).
 - Added `'leverage'` and `'conviction` as evaluation metrics to the `frequent_patterns.association_rules` function. [#246](https://github.com/rasbt/mlxtend/pull/246) & [#247](https://github.com/rasbt/mlxtend/pull/247)
 - Added a `loadings_` attribute to `PrincipalComponentAnalysis` to compute the factor loadings of the features on the principal components. [#251](https://github.com/rasbt/mlxtend/pull/251)
+- Allow grid search over classifiers/regressors in ensemble and stacking estimators [#259](https://github.com/rasbt/mlxtend/pull/259)
 
 ##### Changes
 

docs/sources/user_guide/classifier/EnsembleVoteClassifier.ipynb

@@ -459,6 +459,20 @@
     "grid = grid.fit(iris.data, iris.target)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Note**\n",
+    "\n",
+    "The `EnsembleVoteClass` also enables grid search over the `clfs` argument. However, due to the current implementation of `GridSearchCV` in scikit-learn, it is not possible to search over both, differenct classifiers and classifier parameters at the same time. For instance, while the following parameter dictionary works\n",
+    "\n",
+    "    params = {'randomforestclassifier__n_estimators': [1, 100],\n",
+    "    'clfs': [(clf1, clf1, clf1), (clf2, clf3)]}\n",
+    "    \n",
+    "it will use the instance settings of `clf1`, `clf2`, and `clf3` and not overwrite it with the `'n_estimators'` settings from `'randomforestclassifier__n_estimators': [1, 100]`."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

docs/sources/user_guide/classifier/StackingCVClassifier.ipynb

@@ -423,6 +423,20 @@
     "print('Accuracy: %.2f' % grid.best_score_)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Note**\n",
+    "\n",
+    "The `StackingCVClassifier` also enables grid search over the `classifiers` argument. However, due to the current implementation of `GridSearchCV` in scikit-learn, it is not possible to search over both, differenct classifiers and classifier parameters at the same time. For instance, while the following parameter dictionary works\n",
+    "\n",
+    "    params = {'randomforestclassifier__n_estimators': [1, 100],\n",
+    "    'classifiers': [(clf1, clf1, clf1), (clf2, clf3)]}\n",
+    "    \n",
+    "it will use the instance settings of `clf1`, `clf2`, and `clf3` and not overwrite it with the `'n_estimators'` settings from `'randomforestclassifier__n_estimators': [1, 100]`."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

docs/sources/user_guide/classifier/StackingClassifier.ipynb

@@ -400,6 +400,20 @@
     "print('Accuracy: %.2f' % grid.best_score_)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Note**\n",
+    "\n",
+    "The `StackingClassifier` also enables grid search over the `classifiers` argument. However, due to the current implementation of `GridSearchCV` in scikit-learn, it is not possible to search over both, differenct classifiers and classifier parameters at the same time. For instance, while the following parameter dictionary works\n",
+    "\n",
+    "    params = {'randomforestclassifier__n_estimators': [1, 100],\n",
+    "    'classifiers': [(clf1, clf1, clf1), (clf2, clf3)]}\n",
+    "    \n",
+    "it will use the instance settings of `clf1`, `clf2`, and `clf3` and not overwrite it with the `'n_estimators'` settings from `'randomforestclassifier__n_estimators': [1, 100]`."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

docs/sources/user_guide/regressor/StackingCVRegressor.ipynb

@@ -278,6 +278,20 @@
     "print('Accuracy: %.2f' % grid.best_score_)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Note**\n",
+    "\n",
+    "The `StackingCVRegressor` also enables grid search over the `regressors` argument. However, due to the current implementation of `GridSearchCV` in scikit-learn, it is not possible to search over both, differenct classifiers and classifier parameters at the same time. For instance, while the following parameter dictionary works\n",
+    "\n",
+    "    params = {'randomforestregressor__n_estimators': [1, 100],\n",
+    "    'regressors': [(regr1, regr1, regr1), (regr2, regr3)]}\n",
+    "    \n",
+    "it will use the instance settings of `regr1`, `regr2`, and `regr3` and not overwrite it with the `'n_estimators'` settings from `'randomforestregressor__n_estimators': [1, 100]`."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

docs/sources/user_guide/regressor/StackingRegressor.ipynb

@@ -77,7 +77,9 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "metadata": {},
+   "metadata": {
+    "collapsed": true
+   },
    "outputs": [],
    "source": [
     "from mlxtend.regressor import StackingRegressor\n",
@@ -604,7 +606,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "In case we are planning to use a regression algorithm multiple times, all we need to do is to add an additional number suffix in the parameter grid as shown below:"
+    "**Note**\n",
+    "\n",
+    "The `StackingRegressor` also enables grid search over the `regressors` argument. However, due to the current implementation of `GridSearchCV` in scikit-learn, it is not possible to search over both, differenct classifiers and classifier parameters at the same time. For instance, while the following parameter dictionary works\n",
+    "\n",
+    "    params = {'randomforestregressor__n_estimators': [1, 100],\n",
+    "    'regressors': [(regr1, regr1, regr1), (regr2, regr3)]}\n",
+    "    \n",
+    "it will use the instance settings of `regr1`, `regr2`, and `regr3` and not overwrite it with the `'n_estimators'` settings from `'randomforestregressor__n_estimators': [1, 100]`."
    ]
   },
   {

mlxtend/classifier/ensemble_vote.py

@@ -255,10 +255,7 @@ def get_params(self, deep=True):
 
             for key, value in six.iteritems(super(EnsembleVoteClassifier,
                                             self).get_params(deep=False)):
-                if key == 'clfs':
-                    continue
-                else:
-                    out['%s' % key] = value
+                out['%s' % key] = value
             return out
 
     def _predict(self, X):

mlxtend/classifier/stacking_classification.py

@@ -141,10 +141,7 @@ def get_params(self, deep=True):
 
             for key, value in six.iteritems(super(StackingClassifier,
                                             self).get_params(deep=False)):
-                if key in ('classifiers', 'meta-classifier'):
-                    continue
-                else:
-                    out['%s' % key] = value
+                out['%s' % key] = value
 
             return out
 

mlxtend/classifier/stacking_cv_classification.py

@@ -245,10 +245,7 @@ def get_params(self, deep=True):
 
             for key, value in six.iteritems(super(StackingCVClassifier,
                                             self).get_params(deep=False)):
-                if key in ('classifiers', 'meta-classifier'):
-                    continue
-                else:
-                    out['%s' % key] = value
+                out['%s' % key] = value
 
             return out
 

mlxtend/classifier/tests/test_ensemble_vote_classifier.py

@@ -8,6 +8,7 @@
 from sklearn.linear_model import LogisticRegression
 from sklearn.naive_bayes import GaussianNB
 from sklearn.ensemble import RandomForestClassifier
+from sklearn.neighbors import KNeighborsClassifier
 import numpy as np
 from sklearn import datasets
 from sklearn.model_selection import GridSearchCV
@@ -86,3 +87,38 @@ def test_EnsembleVoteClassifier_gridsearch_enumerate_names():
 
     grid = GridSearchCV(estimator=eclf, param_grid=params, cv=5)
     grid = grid.fit(iris.data, iris.target)
+
+
+def test_get_params():
+    clf1 = KNeighborsClassifier(n_neighbors=1)
+    clf2 = RandomForestClassifier(random_state=1)
+    clf3 = GaussianNB()
+    eclf = EnsembleVoteClassifier(clfs=[clf1, clf2, clf3])
+
+    got = sorted(list({s.split('__')[0] for s in eclf.get_params().keys()}))
+    expect = ['clfs',
+              'gaussiannb',
+              'kneighborsclassifier',
+              'randomforestclassifier',
+              'refit',
+              'verbose',
+              'voting',
+              'weights']
+    assert got == expect, got
+
+
+def test_classifier_gridsearch():
+    clf1 = KNeighborsClassifier(n_neighbors=1)
+    clf2 = RandomForestClassifier(random_state=1)
+    clf3 = GaussianNB()
+    eclf = EnsembleVoteClassifier(clfs=[clf1])
+
+    params = {'clfs': [[clf1, clf1, clf1], [clf2, clf3]]}
+
+    grid = GridSearchCV(estimator=eclf,
+                        param_grid=params,
+                        cv=5,
+                        refit=True)
+    grid.fit(X, y)
+
+    assert len(grid.best_params_['clfs']) == 2