Bayesian Logistic Regression Models for Credit Scoring by Gregg Webster

 
 
3.3.4 Variable selection
 
Variable selection in generalized linear models is often done using a stepwise procedure or 
by best subset selection. Here the stepwise method is introduced for generalized linear 
models (Hosmer and Lemeshow, 2000).
 
Stepwise methods for generalized linear models 
 
The forward stepwise variable selection method starts with no variables in the model and 
adds the most important variables sequentially. The backward stepwise variable selection 
method goes the other way around by starting with a model with all the variables and then 
sequentially deleting variables that provide little value in explaining the response. A 
stepwise procedure is based on a statistical algorithm that checks for the importance of 
variables. The steps are as follows: 
Step 0 (select the best one variable model): Each possible variable is fitted individually 
and compared to the null model using a likelihood ratio test. The 
p
-value for a significant 
variable must fall below a specific significance level. For example, with logistic 
regression, a significance level of between 0.15 and 0.20 is suggested. The variable with 
the smallest 
p
-value below the significance level is chosen.
Step 1 (select the best two variable model): A generalized linear model is fitted containing 
the variable selected in step 0. Models are then fitted using the variable selected in step 0 
and each of the other remaining models. These models are then compared to the model 
with the variable selected in step 0 using a likelihood ratio test. The variable with the 
smallest 
p
-value is then chosen provided it is below the significance level.

---

35 
Step 2: This procedure is continued until all variables are entered into the model or 
additional variables become insignificant.
Alternatively, the backward elimination procedure works by starting with all variables in 
the model, then removing the one that is least significant, then the next, etc until all the 
variables are significant. 
This stepwise algorithm can also be conducted by comparing the AIC (Akaike information 
criterion) instead of using a likelihood ratio test.
These variable selection methods, however, become questionable with binary data. So it is 
better to consider variable selection using expert knowledge about which variables to 
include or not.