Bayesian Logistic Regression Models for Credit Scoring by Gregg Webster

 
Theorem 3.1 
If 
∑
( 
)
( 
)
where 
are i.i.d. from a density 
( )
such 
that 
( )
whenever 
( )
and 
∫ ( )
then 
1.
( ) ∫ ( )
2.
( )
is minimized if 
( )
| ( )|
∫ | ( )|
. 
Proof: 
1.
( ) (
∑
( 
)
( 
)
) (
( )
( )
) ∫
( )
( )
( ) ∫ ( )
2.
From Equation (3.22) 
(∫ | ( )| )
is the lower bound for 
(
( )
( )
)
which is what we 
want to minimize. Now, if 
( )
| ( )|
∫ ( )
(
( )
( )
)
∫ ( ( ))
∫ | ( )|
| ( )|
∫ | ( )|
∫ | ( )|
(∫ | ( )|
)
Thus, for this choice of 
( )
we are at the lower bound and variance is minimized. Now, 
the practical use of Theorem 3.1 is very limited. This is because we need to know the 
integral 
∫ | ( )|
which is for 
( )
the same as 
∫ ( )
. But
∫ ( )
is what 
we are looking to estimate in the first place. This theorem does, however, help us to choose 
a good 
( )
. We should try to achieve 
( )
( )
Therefore, we should sample more 

---

49 
points in regions where 
( )
is large. Thus, the “important” parts of the integral will be 
estimated better. This is why the method is called importance sampling.
 
3.4.2 Markov chains 
An overview of Markov chains is given in this section. The simulation methods described 
previously cannot easily be applied in all cases. Monte Carlo integration and importance 
sampling can be applied when we are dealing with standard distributions. However, when 
we face a non-standard distribution (such as the case with Bayesian logistic regression) the 
previous simulation techniques cannot easily be used to obtain samples from any posterior 
distribution. If they are used, they are subject to major practical difficulties. Markov Chain 
Monte Carlo (MCMC) methods provide a way out.
Markov Chain Monte Carlo methods have greatly improved the scope for Bayesian 
inference (Robert and Casella, 2004; Greenberg, 2008). Because MCMC relies on Markov 
chains, they are now introduced with both discrete and continuous state spaces.