Feature Selection and Variable Selection

Selection objectives

• Predictive selection - Chooses a small or inexpensive input subset for a specified learner and prediction task. Its risk-cost trade-off need not identify a unique set of generating variables. Guyon and Elisseeff 2003.
• Minimal-optimal versus all-relevant - A small subset sufficient for prediction and the full set of relevant variables are different targets; redundant variables can matter to the latter. Nilsson et al. 2007.
• Conditional discovery - Model-X knockoffs test whether the target is independent of a variable given all remaining inputs. This target differs from keeping every useful substitute. Candes et al. 2016.
• Unsupervised selection - Usefulness is defined by a chosen surrogate, such as reconstruction, neighborhoods or cluster structure. Preserving it does not establish utility for an unspecified target. Li et al. 2017.
• Acquisition - Static selection uses one subset; an adaptive policy chooses the next measurement from values already observed. This differs from selecting an explanation after seeing all inputs. Covert et al. 2023.

Interactions and interchangeable variables

• XOR example - For independent fair bits, Y = X1 XOR X2 is independent of either input alone, yet determined by their pair. Even nonlinear univariate dependence scores can miss joint information. Brown et al. 2012.
• Duplicate example - If X1 = X2 = Y, either input predicts perfectly, and each is conditionally uninformative given the other. Changing between these subsets need not change predictions. Statnikov et al. 2013.

Comparative evidence

• Tabular deep-learning benchmark (NeurIPS 2023) - On 12 datasets with MLP/FT-Transformer predictors, the authors report Deep Lasso gains on corrupted and second-order added features. Evidence is specific to those regimes. Cherepanova et al. 2023.
• Added features need not be irrelevant - Products of original variables can help prediction. The 2023 benchmark evaluates downstream quality; rejecting every added column would reward the wrong objective. Cherepanova et al. 2023.
• SelectArena (2026 preprint) - Across 71 datasets and five learners, tree importance and F-tests offer strong prediction/runtime trade-offs; F-tests also rank highly on stability and noise rejection. Schäfer et al. 2026.
• SelectArena protocol - Enforces exactly k features and a one-hour selection budget. Interrupted searches use random filling or subsampling; reported search rankings include this fallback. Schäfer et al. 2026.
• SelectArena metrics - "validity" measures rejection of injected Gaussian-noise columns; it does not establish relevance of every retained original feature. Reported improvement over random subsets uses that comparator. Schäfer et al. 2026.
• Comparison design (editorial guidance) - Use all-feature prediction, random subsets at matched budgets and simple filters as baselines. Report prediction, subset size/cost, selection time and stability separately. Schäfer et al. 2026.

Filter criteria and screening

Marginal screening

• Variance threshold - Removes features with low training-set variance without using labels; scale affects the result. docs.
• Univariate tests - Chi-squared for non-negative counts/frequencies; ANOVA F for class differences; linear-regression F for marginal linear association. Individual scores can miss interactions. docs, docs, docs.
• Sure independence screening (SIS/ISIS) - Screens marginal correlations before subset fitting; ISIS repeats screening. Signal/design assumptions give high-probability retention of all signals, while extra variables may remain. Fan and Lv 2006.

Information, redundancy, and interactions

• Mutual information - A dependence quantity; Kraskov et al. provide an estimator. Selection criteria include mRMR/JMI, while forward search constructs a subset. These are separate choices. Kraskov et al. 2003, Brown et al. 2012.
• MI estimation - Discrete and continuous features require appropriate treatment; mislabeling their type can give incorrect estimates. Nonlinear dependence estimation does not make marginal scoring interaction-aware. docs.
• mRMR - Balances feature-target mutual information against redundancy among selected features. Hanchuan Peng et al. 2005.
• Joint mutual information (JMI) - Scores candidates jointly with already selected variables. Brown et al. derive JMI and related criteria as approximations to conditional-likelihood selection. Yang and Moody 1999, Brown et al. 2012.
• FCBF (fast correlation-based filter) - Uses symmetrical uncertainty to screen relevance and remove redundant features. Yu and Liu 2003.
• Relief - Updates feature weights from nearby same-class and different-class examples. Relief-family review covers interaction sensitivity and extensions. Kira and Rendell 1992, Urbanowicz et al. 2017.
• Consistency-based selection - Penalizes identical selected-feature patterns with conflicting labels. Dash and Liu compare search strategies using this criterion. Dash and Liu 2003.
• INTERACT - Removes features using changes in class inconsistency, targeting interactions that marginal ranking can miss. Zhao and Liu 2007.

Conditional independence and Markov boundaries

• FOCI - Greedily adds variables using a conditional-dependence statistic. Its sufficient-set recovery result needs detectable improvement at every insufficient subset; purely joint signals can defeat this forward-search condition. Azadkia and Chatterjee 2019.
• Markov blankets - Seek variables that render the target conditionally independent of the remaining variables. Discovery depends on conditional-independence tests and distributional assumptions. Fu and Desmarais 2010.
• Markov boundary - An inclusion-minimal blanket. Multiple boundaries can exist, yielding alternative sufficient subsets; disagreement need not identify a uniquely wrong selection. Statnikov et al. 2013.
• Koller-Sahami - Eliminates features using approximate Markov blankets and information-theoretic redundancy criteria. Koller and Sahami 1996.
• Grow-Shrink - Grows a candidate blanket and then removes conditionally irrelevant variables. Margaritis and Thrun 1999.
• IAMB - Incremental association-based blanket discovery, with forward inclusion and backward pruning. Tsamardinos et al. 2003.
• Max-Min blanket discovery - Identifies parents/children and extends the search to a target's Markov blanket. A predictive blanket alone does not identify causal direction. Tsamardinos et al. 2003.

Subset search and cardinality optimization

• Wrappers - Evaluate candidate subsets using a particular learner's predictive performance. Search and validation are part of the method. Kohavi and John 1997.
• Sequential selection - Greedily adds or removes features according to cross-validation scores. SequentialFeatureSelector supports forward and backward search. docs.
• Floating search - Alternates inclusion and exclusion to revisit earlier choices. Adaptive floating search varies the search depth. Pudil et al. 1994, Somol et al. 1999.
• Recursive feature elimination (RFE) - Repeatedly fits a model and removes low-ranked inputs. RFE and RFECV implement elimination and cross-validated subset-size selection. Guyon et al. 2002; docs, docs.
• Population-based subset search - Genetic search and particle swarms explore subsets scored by a chosen criterion; that criterion defines the selection objective. Miao and Niu 2016, Kennedy and Eberhart 1995, Tu et al. 2007.
• Best-subset optimization - Mixed-integer formulations constrain feature count directly. Solver bounds can certify global optimality or quantify a remaining gap when stopped early. Bertsimas et al. 2015.
• Fast L0 selection - Coordinate descent and local combinatorial search optimize L0-penalized objectives. Their local optimality conditions differ from a global MIO certificate. Hazimeh and Mazumder 2018.

Embedded sparsity and model-based selection

Penalized and Bayesian models

• Lasso - An L1 penalty yields sparse regression coefficients, coupling variable selection with fitting. Tibshirani 1996.
• Adaptive lasso - Weights each coefficient's L1 penalty using an initial estimate. Zou establishes selection consistency and oracle asymptotics under design, initialization and tuning conditions; this is not a finite-sample guarantee of correct support. Zou 2006.
• Lasso support recovery - Sparse prediction does not establish correct variable recovery. Selection consistency requires additional assumptions, notably restrictions on predictor dependence. Zhao and Yu 2006.
• Elastic net - Combines L1 and L2 penalties, encouraging groups of correlated predictors to enter together. Zou and Hastie 2005.
• Group lasso - Selects predefined groups of coefficients, such as all encoded levels of a categorical variable. This differs from elastic net's grouping of correlated predictors. Yuan and Lin 2005.
• Spike-and-slab - Mixture priors represent exclusion and inclusion of regression coefficients. George and McCulloch develop computational approaches. Mitchell and Beauchamp 1988, George and McCulloch 1997.
• HSIC Lasso - Uses feature-wise kernels to select variables with strong nonlinear dependence on the response and limited redundancy with one another; a sparse alternative to linear Lasso screening. Yamada et al. 2014.

Trees, importance, and subset rules

• Decision trees - Split-based importance sums impurity reductions attributed to each input; thresholding can select a subset. These scores can favor high-cardinality inputs. docs, docs.
• Random forests - Ensembles provide importance scores used for ranking and selection. Boulesteix et al. review practical pitfalls. A ranking still needs a subset-selection rule. Breiman 2001, Boulesteix et al. 2012.
• Gradient boosting - Sequential function fitting can supply feature-importance rankings. Importance thresholding produces a subset. Friedman 2001, Mason et al. 1999; docs.
• Boruta - Targets all-relevant selection by comparing random-forest importances with randomized shadow features. Kursa et al. 2010, Kursa and Rudnicki 2010; code.
• VSURF - Forest importance and stepwise selection produce subsets for interpretation and prediction. Genuer et al. 2010, Genuer et al. 2015.

Differentiable selectors

• Concrete autoencoders - Learn which original inputs to retain through a relaxed discrete selector and reconstruction decoder. Balın et al. 2019; code.
• Stochastic gates (STG) - Learn input gates jointly with a predictor using a continuous relaxation of an L0 feature-count penalty. Yamada et al. 2020; code.
• LassoNet - Ties nonlinear feature use to sparse linear skip connections through a hierarchical constraint; produces a regularization path of global subsets. Lemhadri et al. 2021; code.
• Deep Lasso - Applies a group penalty to loss gradients with respect to input features, extending linear sparsity to input sensitivity. Gradient norms also rank features. Cherepanova et al. 2023.

Unsupervised and graph-based selection

• FSSEM - Searches feature subsets alongside expectation-maximization clustering; examines likelihood and scatter criteria and their dimensionality biases. Dy and Brodley 2004.
• Laplacian Score - Ranks features by preservation of local graph structure; applicable without class labels. He et al. 2005.
• Principal feature analysis - Uses PCA-derived structure to choose representative original variables. Its output is an input subset rather than principal-component coordinates. Lu et al. 2007.
• Spectral feature selection (SPEC) - Evaluates features through a similarity graph's spectrum, covering supervised and unsupervised criteria. Zhao and Liu 2007.
• Multi-cluster feature selection (MCFS) - Combines a spectral embedding with sparse regression to select features preserving cluster structure. Cai et al. 2010.

Resampling, error control and post-selection inference

• Stability selection - Thresholds selection frequencies across subsamples. Its original bound controls the expected number of false selections under exchangeability and other assumptions. Meinshausen and Buehlmann 2008.
• Complementary-pairs stability selection - Pairs disjoint half-samples and bounds selection of variables with low selection probabilities, without the original exchangeability requirement. Shah and Samworth 2012.
• Knockoff+ - Selects against synthetic control variables with finite-sample false discovery rate control in the paper's Gaussian linear-model setting. Barber and Candès 2015.
• Model-X knockoffs - Permits arbitrary response models; FDR control requires exchangeability and knockoffs independent of the response conditional on the inputs. Candes et al. 2016; docs.
• Knockoff robustness - Flexibility in the response model relies on modeling the input distribution. Errors in estimated feature conditionals can inflate FDR; robustness bounds quantify this dependence. Barber et al. 2018.
• Artificial controls, different targets - Boruta compares forest importances with shuffled shadows for all-relevant selection; model-X uses exchangeable controls for conditional discovery with FDR control under its assumptions. Kursa and Rudnicki 2010, Candes et al. 2016.
• Ensemble selection - Aggregates feature preferences across perturbed datasets or selectors. Aggregation alone supplies no generic FDR guarantee. Saeys et al. 2008, Shen et al. 2020.
• Deep knockoffs - Learns approximate model-X knockoff generators by encouraging pairwise exchangeability. Generator diagnostics matter: approximate sampling alone does not establish exact false-discovery control. Romano et al. 2019.
• Post-selection inference - Conditions on lasso selection to form intervals for selected-model coefficients. Exact results use a fixed design and Gaussian errors with known variance. Lee et al. 2016.

Instancewise selection

• Instancewise selection - L2X learns a feature subset for each example to explain a fixed predictor; this differs from one global subset used for all observations. Chen et al. 2018.
• INVASE - A selector, predictor and baseline network learn instance-specific feature subsets through actor-critic training; the selected variables and subset size can change between observations. Yoon et al. 2019.

Cost-sensitive and adaptive acquisition

• Dynamic feature selection - Learns a policy approximating greedy conditional-mutual-information acquisition from observed inputs. Recovery of the greedy policy assumes optimal training; it is not a globally optimal acquisition guarantee. Covert et al. 2023.

Evaluation and comparative evidence

Validation and selection stability

• Selection leakage - Choosing features before resampling can bias estimated prediction error. Ambroise and McLachlan demonstrate the problem in microarray classification; pipeline guidance keeps selection inside training folds. Ambroise and McLachlan 2002; docs.
• Search overfitting - A noisy validation criterion can overfit selector, subset-size and predictor choices. Nested resampling or an untouched test set must evaluate the complete search, including learned preprocessing. Cawley and Talbot 2010.
• MUVR - Combines variable selection with repeated double cross-validation in multivariate modeling. Shi et al. 2018; code.
• Stability measurement - Compares scores, ranks, or subsets under data perturbations. Nogueira, Sechidis, Brown provide a chance-corrected measure, intervals, and tests. Kalousis et al. 2006, Nogueira et al. 2018.

Correlated features and importance bias

• Forest importance bias - Strobl et al. show preference for variables with more categories or different measurement scales in studied forest importance measures. Importance is not an unbiased relevance test. Strobl et al. 2007.
• Conditional permutation importance - Permutes within a conditioning scheme to account for predictor dependence. Gregorutti et al. analyze correlation effects and recursive elimination. Strobl et al. 2008, Gregorutti et al. 2013.
• Held-out permutation importance - Measures a fitted model's score loss after permuting a feature. Correlated substitutes can mask importance; the result depends on the model and score. docs.

Task-specific selection and method recommendation

• Uplift modeling - Selects inputs for heterogeneous treatment-effect prediction rather than ordinary outcome prediction. Zhao et al. develop task-specific statistical and information-theoretic criteria. Zhao et al. 2022.
• Selector recommendation - Learns from dataset metafeatures and previous selector evaluations. Wang et al., Parmezan et al., and synthetic metatraining study this AutoML connection. Wang et al. 2014, Parmezan et al. 2017, Shen et al. 2020.

Software

• Python (classical selection) - scikit-learn provides filters, model thresholds, RFE/RFECV, and sequential selection; scikit-feature collects research implementations. docs, code.
• Python (learned selection and error control) - Concrete autoencoders, STG, LassoNet, knockpy. code, code, code, docs.
• Supporting tools - feature-selector removes missingness/constant/collinearity-based columns; these are data-quality rules. BoostARoota implements XGBoost-based shadow comparisons. code, code.
• R (search and filters) - FSinR and praznik; copent supplies supporting copula/transfer-entropy estimation. Aragón-Royón et al. 2020, Ma 2020; code, code, code.
• R (model-based selection) - Boruta, VSURF, glmnet, spikeSlabGAM, knockoff. Scheipl 2011; code, code, code, code, code.
• L0Learn - Approximate L0-regularized regression and classification through coordinate descent and local combinatorial search; C++ implementation with R/Python interfaces. Hazimeh et al. 2023.
• Archived software - fscaret was archived by CRAN in August 2025; retained for historical reference. code.

Surveys and further reading

• An Introduction to Variable and Feature Selection - Relevance, redundancy, ranking, subset search, and validation. Guyon and Elisseeff 2003.
• A Survey on Feature Selection - Filter, wrapper, and embedded approaches. Miao and Niu 2016.
• Feature Selection: A Data Perspective - Conventional, structured, heterogeneous, and streaming data. Li et al. 2017.
• Feature Selection and Feature Extraction in Pattern Analysis - Selection of existing variables and construction of new representations. Ghojogh et al. 2019.
• Attribution rankings - Man-Chan compare permutation, LIME and SHAP rankings for random forests; reported stability differences do not establish predictive gains from selecting those features. Man and Chan 2020.

Connections

• Representation learning - Ordinary autoencoders construct latent features. Concrete autoencoders instead constrain the encoder to select original inputs. Baldi 2012, Kingma and Welling 2019, Balın et al. 2019.

Related topics

• Feature extraction
• Interpretability
• Causal inference
• Ensemble learning
• AutoML