Target mean selection closes #170 (#181)

* created target mean encoding

* created test target mean encoding

* update target mean encoding feature selector

* update target mean encoding feature selector (2)

* update target mean encoding feature selector (2)

* update target mean encoding feature selector (3)

* update target mean encoding feature selector (3)

* update target mean encoding feature selector (3)

* refractor select by target mean

* rename feature shuffling selector

* fix docstrings mean_target_selection

* add cv, fixes bugs, adds 1 test

* finish select with target meatn

* docs for select with target mean encoding

* add contributor to changelog

Co-authored-by: ttungl <[email protected]>

Author: solegalli

README.md

@@ -99,6 +99,8 @@ More resources will be added as they appear online!
  * DropDuplicateFeatures
  * DropCorrelatedFeatures
  * ShuffleFeaturesSelector
+ * SelectBySingleFeaturePerformance
+ * SelectByTargetMeanPerformance
  * RecursiveFeatureElimination
 
 
@@ -125,7 +127,7 @@ git clone https://github.com/solegalli/feature_engine.git
 ### Usage
 
 ```python
->>> from feature_engine.categorical_encoders import RareLabelCategoricalEncoder
+>>> from feature_engine.encoding import RareLabelEncoder
 >>> import pandas as pd
 
 >>> data = {'var_A': ['A'] * 10 + ['B'] * 10 + ['C'] * 2 + ['D'] * 1}
@@ -143,7 +145,7 @@ Name: var_A, dtype: int64
 ```
 
 ```python 
->>> rare_encoder = RareLabelCategoricalEncoder(tol=0.10, n_categories=3)
+>>> rare_encoder = RareLabelEncoder(tol=0.10, n_categories=3)
 >>> data_encoded = rare_encoder.fit_transform(data)
 >>> data_encoded['var_A'].value_counts()
 ```

docs/changelog.rst

@@ -12,6 +12,7 @@ Contributors:
     - Sana Ben Driss
     - Nicolas Galli
     - Tejash Shah
+    - Tung Lee
     - Soledad Galli
 
 
@@ -52,13 +53,15 @@ We renamed a few parameters to unify the nomenclature across the Package.
     - **DropDuplicateFeatures**: DropDuplicateFeatures finds and removes duplicated features from a dataset (**by Tejash Shah and Soledad Galli**)
     - **DropCorrelatedFeatures**: DropCorrelatedFeatures finds and removes features that are correlated (**by Nicolas Galli**)
     - **ShuffleFeaturesSelector**: ShuffleFeaturesSelector selects features by determining the drop in machine learning model performance when each feature's values are randomly shuffled from a dataframe (**by Sana Ben Driss**)
+    - **SelectBySingleFeaturePerformance**: SelectBySingleFeaturePerformance trains a model based of each individual features, and derives performance (**by Nicolas Galli**)
+    - **SelectByTargetMeanPerformance**: SelectByTargetMeanPerformance selects features encoding the categories with the target mean and using that as proxy for performance (**by Tung Lee and Soledad Galli**)
     - **RecursiveFeatureElimination**: RecursiveFeatureElimination selects features recursively, evaluating the drop in ML performance, from the least to the important feature (**by Sana Ben Driss**)
 
 **Code Architecture - Important for Contributors and Developers**:
     - **Submodules**: transformers have been grouped within relevant submodules and modules.
     - **Individual tests**: testing classes have been subdivided into individual tests
     - **Code Style**: we adopted the use of flake8 for linting and PEP8 style checks, and black for automatic re-styling of code.
-    - **Type hint**: we are slowly rolling out the use of type hint throughout Feature-engine classes and functions (**by Nodar Okroshiashvili**)
+    - **Type hint**: rolled out the use of type hint throughout Feature-engine classes and functions (**by Nodar Okroshiashvili, Soledad Galli and Chris Samiullah**)
 
 **Other Changes**:
     - **Updated documentation**: documentation reflects the current use of Feature-engine transformers

docs/index.rst

@@ -177,6 +177,7 @@ Feature Selection:
 - :doc:`selection/DropCorrelatedFeatures`: drops correlated variables from a dataframe
 - :doc:`selection/ShuffleFeaturesSelector`: selects features by evaluating model performance after feature shuffling
 - :doc:`selection/SelectBySingleFeaturePerformance`: selects features based on their performance on univariate estimators
+- :doc:`selection/SelectByTargetMeanPerformance`: selects features based on target mean encoding performance
 - :doc:`selection/RecursiveFeatureElimination`: selects features recursively, by evaluating model performance
 
 Getting Help

docs/selection/SelectByTargetMeanPerformance.rst

@@ -0,0 +1,13 @@
+SelectByTargetMeanPerformance
+=============================
+
+The SelectByTargetMeanPerformance()selects features based on the performance of
+machine learning models trained using individual features. In other words, selects
+features based on their individual performance, returned by estimators trained on
+only that particular feature.
+
+API Reference
+-------------
+
+.. autoclass:: feature_engine.selection.SelectByTargetMeanPerformance
+    :members:

docs/selection/ShuffleFeaturesSelector.rst

@@ -1,7 +1,7 @@
-ShuffleFeaturesSelector
-=======================
+SelectByShuffling
+=================
 
-The ShuffleFeaturesSelector() selects important features if permutation their values
+The SelectByShuffling() selects important features if permutation their values
 at random produces a decrease in the initial model performance. See API below for
 more details into its functionality.
 
@@ -21,7 +21,7 @@ more details into its functionality.
     linear_model = LinearRegression()
 
     # initialize feature selector
-    tr = ShuffleFeaturesSelector(estimator=linear_model, scoring="r2", cv=3)
+    tr = SelectByShuffling(estimator=linear_model, scoring="r2", cv=3)
 
     # fit transformer
     Xt = tr.fit_transform(X, y)
@@ -75,5 +75,5 @@ more details into its functionality.
 API Reference
 -------------
 
-.. autoclass:: feature_engine.selection.ShuffleFeaturesSelector
+.. autoclass:: feature_engine.selection.SelectByShuffling
     :members:

docs/selection/index.rst

@@ -15,4 +15,5 @@ Or in other words to select subsets of variables.
    DropCorrelatedFeatures
    ShuffleFeaturesSelector
    SelectBySingleFeaturePerformance
+   SelectByTargetMeanPerformance
    RecursiveFeatureElimination

feature_engine/selection/__init__.py

@@ -5,16 +5,18 @@
 from .drop_constant_features import DropConstantFeatures
 from .drop_duplicate_features import DropDuplicateFeatures
 from .drop_correlated_features import DropCorrelatedFeatures
-from .shuffle_features import ShuffleFeaturesSelector
-from .single_feature_performance_selection import SelectBySingleFeaturePerformance
+from .shuffle_features import SelectByShuffling
+from .single_feature_performance import SelectBySingleFeaturePerformance
 from .recursive_feature_elimination import RecursiveFeatureElimination
+from .target_mean_selection import SelectByTargetMeanPerformance
 
 __all__ = [
     "DropFeatures",
     "DropConstantFeatures",
     "DropDuplicateFeatures",
     "DropCorrelatedFeatures",
-    "ShuffleFeaturesSelector",
+    "SelectByShuffling",
     "SelectBySingleFeaturePerformance",
     "RecursiveFeatureElimination",
+    "SelectByTargetMeanPerformance",
 ]

feature_engine/selection/shuffle_features.py

@@ -20,18 +20,18 @@
 Variables = Union[None, int, str, List[Union[str, int]]]
 
 
-class ShuffleFeaturesSelector(BaseEstimator, TransformerMixin):
+class SelectByShuffling(BaseEstimator, TransformerMixin):
     """
 
-    ShuffleFeaturesSelector selects features by determining the drop in machine learning
+    SelectByShuffling selects features by determining the drop in machine learning
     model performance when each feature's values are randomly shuffled.
 
     If the variables are important, a random permutation of their values will
     decrease dramatically the machine learning model performance. Contrarily, the
     permutation of the values should have little to no effect on the model performance
     metric we are assessing.
 
-    The ShuffleFeaturesSelector first trains a machine learning model utilising all
+    The SelectByShuffling first trains a machine learning model utilising all
     features. Next, it shuffles the values of 1 feature, obtains a prediction with the
     pre-trained model, and determines the performance drop (if any). If the drop in
     performance is bigger than a threshold then the feature is retained, otherwise
@@ -116,6 +116,7 @@ def __init__(
 
     def fit(self, X: pd.DataFrame, y: pd.Series):
         """
+        Finds the important features
 
         Args
         ----

…/single_feature_performance_selection.py …/selection/single_feature_performance.pyfeature_engine/selection/single_feature_performance_selection.py renamed to feature_engine/selection/single_feature_performance.py feature_engine/selection/single_feature_performance_selection.py renamed to feature_engine/selection/single_feature_performance.py

@@ -93,9 +93,11 @@ def __init__(
         if not isinstance(threshold, (int, float)):
             raise ValueError("threshold can only be integer or float")
 
-        if scoring == 'roc_auc' and (threshold < 0.5 or threshold > 1):
-            raise ValueError("roc-auc score should vary between 0.5 and 1. Pick a "
-                             "threshold within this interval.")
+        if scoring == "roc_auc" and (threshold < 0.5 or threshold > 1):
+            raise ValueError(
+                "roc-auc score should vary between 0.5 and 1. Pick a "
+                "threshold within this interval."
+            )
 
         self.variables = _check_input_parameter_variables(variables)
         self.estimator = estimator
@@ -151,9 +153,7 @@ def fit(self, X: pd.DataFrame, y: pd.Series):
 
         # check we are not dropping all the columns in the df
         if len(self.selected_features_) == 0:
-            raise ValueError(
-                "No features were selected, try changing the threshold."
-            )
+            raise ValueError("No features were selected, try changing the threshold.")
 
         self.input_shape_ = X.shape