step-select

A SciKit-Learn style feature selector using best subsets and stepwise regression.

Install

Create a virtual environment with Python 3.8 and install from PyPi:

bash pip install step-select

Use

Preliminaries

Note: this example requires two additional packages: pandas and statsmodels.

In this example we'll show how the ForwardSelector and SubsetSelector classes can be used on their own or in conjuction with a Scikit-Learn Pipeline object.

```python import pandas as pd from sklearn.pipeline import Pipeline from sklearn.linearmodel import LinearRegression import statsmodels.datasets from statsmodels.api import OLS from statsmodels.tools import addconstant

from steps.forward import ForwardSelector from steps.subset import SubsetSelector ```

We'll download the auto dataset via Statsmodels; we'll use mpg as the endogenous variable and the remaining variables as exongenous. We won't use make, as that will create several dummies and increase the number of paramters to 12+, which is too many for the SubsetSelector class; we'll also drop price.

python data = statsmodels.datasets.webuse('auto') data['foreign'] = pd.Series([x == 'Foreign' for x in data['foreign']]).astype(int) data.fillna(0, inplace=True) data.head()

python X = data.iloc[:, 3:] y = data['mpg']

Forward Stepwise Selection

The ForwardSelector follows the standard stepwise regression algorithm: begin with a null model, iteratively test each variable and select the one that gives the most statistically significant improvement of the fit, and repeat. This greedy algorithm continues until the fit no longer improves.

The ForwardSelector is instantiated with two parameters: normalize and metric. Normalize defaults to False, assuming that this class is part of a larger pipeline; metric defaults to AIC.

|Parameter|Type|Description| |---------|----|-----------| |normalize|bool|Whether to normalize features; default False| |metric|str|Optimization metric to use; must be one of aic or bic; default aic|

The ForwardSelector class follows the Scikit-Learn API. After fitting the selector using the .fit() method, the selected features can be accessed using the boolean mask under the .best_support_ attribute.

python selector = ForwardSelector(normalize=True, metric='aic') selector.fit(X, y)

ForwardSelector(normalize=True)

python X.loc[:, selector.best_support_]

Best Subset Selection

The SubsetSelector follows a very simple algorithm: compare all possible models with $k$ predictors, and select the model that minimizes our selection criteria. This algorithm is only appropriate for $k<=12$ features, as it becomes computationally expensive: there are $\frac{k!}{(p-k)!}$possible models, where $p$ is the total number of paramters and $k$ is the number of features included in the model.

The SubsetSelector is instantiated with two parameters: normalize and metric. Normalize defaults to False, assuming that this class is part of a larger pipeline; metric defaults to AIC.

|Parameter|Type|Description| |---------|----|-----------| |normalize|bool|Whether to normalize features; default False| |metric|str|Optimization metric to use; must be one of aic or bic; default aic|

The SubsetSelector class follows the Scikit-Learn API. After fitting the selector using the .fit() method, the selected features can be accessed using the boolean mask under the .best_support_ attribute.

python selector = SubsetSelector(normalize=True, metric='aic') selector.fit(X, y)

SubsetSelector(normalize=True)

python X.loc[:, selector.get_support()]

Comparing the full model

Using the SubsetSelector selected features yields a model with 4 fewer parameters and slightly improved AIC and BIC metrics. The summaries indicate possible multicollinearity in both models, likely caused by weight, length, displacement and other features that are all related to the weight of a vehicle.

Note: Selection using BIC as the optimization metric yields a model where weight is the only selected feature. Bayesian information criteria penalizes additional parameters more then AIC.

python mod = OLS(endog=y, exog=add_constant(X)).fit() mod.summary()

OLS Regression Results

Dep. Variable:	mpg	R-squared:	0.720
Model:	OLS	Adj. R-squared:	0.681
Method:	Least Squares	F-statistic:	18.33
Date:	Sat, 07 Aug 2021	Prob (F-statistic):	1.29e-14
Time:	15:37:36	Log-Likelihood:	-187.23
No. Observations:	74	AIC:	394.5
Df Residuals:	64	BIC:	417.5
Df Model:	9
Covariance Type:	nonrobust

	coef	std err	t	P>|t|	[0.025	0.975]
const	39.0871	9.100	4.295	0.000	20.907	57.267
rep78	1.0021	0.357	2.809	0.007	0.290	1.715
headroom	-0.0167	0.611	-0.027	0.978	-1.237	1.204
trunk	-0.0772	0.154	-0.503	0.617	-0.384	0.230
weight	-0.0037	0.002	-1.928	0.058	-0.008	0.000
length	-0.0752	0.061	-1.229	0.223	-0.197	0.047
turn	-0.1762	0.187	-0.941	0.350	-0.550	0.198
displacement	0.0131	0.011	1.180	0.243	-0.009	0.035
gear_ratio	3.7067	1.751	2.116	0.038	0.208	7.206
foreign	-4.4633	1.385	-3.222	0.002	-7.230	-1.696

Omnibus:	28.364	Durbin-Watson:	2.523
Prob(Omnibus):	0.000	Jarque-Bera (JB):	52.945
Skew:	1.389	Prob(JB):	3.18e-12
Kurtosis:	6.074	Cond. No.	7.55e+04

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 7.55e+04. This might indicate that there are
strong multicollinearity or other numerical problems. ```python mod = OLS(endog=y, exog=add_constant(X.loc[:, selector.best_support_])).fit() mod.summary() ```

OLS Regression Results

Dep. Variable:	mpg	R-squared:	0.710
Model:	OLS	Adj. R-squared:	0.688
Method:	Least Squares	F-statistic:	33.25
Date:	Sat, 07 Aug 2021	Prob (F-statistic):	5.22e-17
Time:	15:37:40	Log-Likelihood:	-188.63
No. Observations:	74	AIC:	389.3
Df Residuals:	68	BIC:	403.1
Df Model:	5
Covariance Type:	nonrobust

	coef	std err	t	P>|t|	[0.025	0.975]
const	40.3703	7.860	5.136	0.000	24.687	56.054
rep78	0.9040	0.342	2.647	0.010	0.223	1.586
weight	-0.0030	0.002	-1.770	0.081	-0.006	0.000
length	-0.1058	0.053	-1.990	0.051	-0.212	0.000
gear_ratio	2.6905	1.511	1.780	0.079	-0.325	5.706
foreign	-4.0123	1.320	-3.040	0.003	-6.646	-1.379

Omnibus:	24.257	Durbin-Watson:	2.442
Prob(Omnibus):	0.000	Jarque-Bera (JB):	39.774
Skew:	1.252	Prob(JB):	2.31e-09
Kurtosis:	5.576	Cond. No.	6.59e+04

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 6.59e+04. This might indicate that there are
strong multicollinearity or other numerical problems.

Use in Scikit-Learn Pipeline

Both ForwardSelector and SubsetSelector objects are compatible with Scikit-Learn Pipeline objects, and can be used as feature selection steps:

python pl = Pipeline([ ('feature_selection', SubsetSelector(normalize=True)), ('regression', LinearRegression()) ]) pl.fit(X, y)

Pipeline(steps=[('feature_selection', SubsetSelector(normalize=True)),
                ('regression', LinearRegression())])

python pl.score(X, y)

0.7097132531085899