Bayesian Logistic Regression Models for Credit Scoring by Gregg Webster

 
 

---

46 
Monte Carlo integration
 
Monte Carlo integration is a statistical technique for approximating integrals. It uses 
simulation to obtain an estimate of the integral which has a mean and a variance. One 
method of Monte Carlo Integration is the sample mean approach. This method is described 
below for the estimation of the integral, 
∫ ( )
. The following approach is 
discussed in Suess and Trumbo (2010). 
Now, if 
( )
then
( ( )) ∫ (
) ( )
∫ ( )
. Therefore, 
∫ ( ) ( ) ( ( )) 
The integral 
∫ ( )
can, therefore, be approximated by 
∑
( 
)
(3.19) 
where 
are random numbers from 
( )
. The mean and variance of this 
estimator is derived as follows: 
( ) (
∑ ( 
))
( ( ))
∫ ( )
Therefore, the estimator in Equation (3.19) is an unbiased estimator for the integral, 
∫ ( )
. Now, for the variance 
( ) (
∑ ( 
)) 
( )
( ( )) 
( )
( ( ( )
) ( ( ( )))
( )
∫ ( ( ))
(
∫ ( )
)
( ) ∫ ( ( ))
(∫ ( )
)
(3.20) 

---

47 
So, we have that 
( )
.
Importance sampling 
Importance sampling is used to reduce the variance of a Monte Carlo estimate of an 
integral. From Equation (3.20) the standard deviation of an estimator for the integral, 
is 
( )
√ 
. Thus, the standard deviation of the estimator decreases as 
increases, but at 
a decreasing rate. This means that if we increase the number of random points from 
to 
points, the standard deviation is improved from the order of 
to 
. 
Therefore, quite a large number of random points are needed to obtain a noticeable 
improvement in accuracy. Importance Sampling aims to improve the standard deviation of 
a Monte-Carlo estimate. The idea is as follows as seen in Robert and Casella (2004). 
Consider a density 
( )
on 
with the property that 
( )
whenever 
( )
. 
Then
∫ ( ) ∫
( )
( )
( )
(
( )
( )
)
if 
( )
. 
Therefore, in order to obtain an estimate for 
∫ ( )
using importance sampling, we 
sample 
from 
( )
and estimate
∫ ( )
∑
( 
)
( 
)
(3.21) 
The new estimator is given by 
∑
( 
)
( 
)
and the variance of 
is given by 
( ) (
∑
( 
)
( 
)
) 
(
( )
( )
) 
[ (
( )
( )
)
( (
( )
( )
))
] 
[∫
( ( ))
( ( ))
( ) (∫ ( ) )
)
]

---

48 
To minimize the variance, we need to minimize the first term 
(
( )
( )
)
. Using Jensen’s 
inequality 
(
( )
( )
)
(
| ( )|
( )
) 
(∫
| ( )|
( )
( ) )
(∫ | ( )| )
(3.22) 
which is a lower bound and does not depend on the choice of 
( )
.