Data Analysis Using Python > Data Processing > Feature Selection

 

1. Filter Methods

• Variance Threshold: Removes features whose variance doesn't meet a specific threshold.
• Univariate Selection: Uses statistical tests (like Chi-Square, ANOVA F-test, or Mutual Information) to select the K highest-scoring features.

            from sklearn.feature_selection import SelectKBest, chi2

           selector = SelectKBest(score_func=chi2, k=2)

           X_new = selector.fit_transform(X, y)

2. Wrapper Methods

Wrapper methods follow the selection process as a search problem. These methods train a particular machine learning estimator on different combinations of features and evaluate performance to add or remove columns.

• Sequential Feature Selector (SFS): A greedy algorithm that adds features one by one (Forward Selection) or removes them one by one (Backward Elimination).
• Recursive Feature Elimination (RFE): Fits a model, ranks features by importance, and recursively prunes the least important features until the target subset size is reached.

     from sklearn.linear_model import LogisticRegression

     X, y = load_iris(return_X_y=True)

     estimator = LogisticRegression(max_iter=200)

     rfe_selector = RFE(estimator=estimator, n_features_to_select=2)

     rfe_selector.fit(X, y)

     print("Feature Support:", rfe_selector.support_)

     print("Feature Ranking:", rfe_selector.ranking_)

 

3. Embedded Methods

Embedded methods perform feature selection internally as part of the model's training process.

• L1 Regularization (Lasso): Adds a penalty that forces the weights of irrelevant features to become exactly zero.
• Tree-based Importance: Algorithms like Random Forests and Gradient Boosting natively calculate feature importance scores based on how much they reduce impurity.

        from sklearn.ensemble import RandomForestClassifier

        from sklearn.feature_selection import SelectFromModel

        X, y = load_iris(return_X_y=True)

       clf = RandomForestClassifier(n_estimators=50, random_state=42)

       sfm_selector = SelectFromModel(estimator=clf)

       X_new = sfm_selector.fit_transform(X, y)

       print("Features kept after tree-based selection:", X_new.shape[1])

       

       

Prev

Recoding using Label Encoder

No More

Feedback

Submit