Bayesian Logistic Regression Models for Credit Scoring by Gregg Webster

3.4.1 Monte Carlo simulation 
 
In Bayesian inference, simulation is needed to evaluate integrals. In order to do this, it is 
essential that random data can be generated. The generation of random variables and all 
other Monte Carlo methods are reliant on the generation of uniform random variables on 
the interval 
( )
. 
 
Uniform random number generation 
 
There are many methods to produce pseudo uniform random numbers as shown in Kroese 
et al
. (2011). These generators include Linear congruential, Multiple-recursive, Matrix 
congruential, Modulo 2 linear etc. The function for the multiple-recursive generator is as 
follows: 
 
where 
and 
are positive integers and 
means that 
is divided by 
and 
the remainder is taken as the next value 
. To use the generator, only a starting number 

---

43 
is thus needed. This starting number is called the seed. Once the desired number of random 
numbers have been generated, each number is divided by 
. This results in uniform 
random numbers on the interval 
( )
.
According to Kroese 
et al
. (2011) two excellent generators that have very good 
performance are: 
-
Combined multiple-recursive generators.
-
Twisted general feedback shift register generators.
Luckily, these very good generators are what are used in computer programs and statistical 
software. For example the program MATLAB uses the twisted general feedback shift 
register generator.
 
Random variable generation
 
Two common methods for random variable generation are the inverse transform method 
and accept-reject algorithm.
Inverse-transform method 
Kroese 
et al
. (2011) introduces the inverse-transform method as follows:
 
Let 
be a random variable with cumulative distribution function (cdf) 
( ) ( )
. 
Since 
is a non-decreasing function, the inverse function can be defined as 
( ) {
{ ( ) }
Now if we have a random variable 
from a uniform distribution on 
( )
, i.e 
( ) 
then the cdf of the inverse transform 
( )
is given by 
( 
( ) ) ( ( )) ( ) 

---

44 
Thus, in order to generate a random variable X with cumulative distribution function 
( )
, 
we generate U from 
( ) 
and then make the transformation 
( )
. Therefore, 
we have the following algorithm 
-
1. Generate 
from 
( ) 
-
2. Return 
( ) 
This is used for sampling random variables from continuous distributions. Obviously, this 
method only works when we can determine and evaluate the inverse of the cdf 
.
Accept-reject method 
The inverse transform method is of no use when one cannot obtain the inverse of the 
cumulative distribution function. A more general method is the accept-reject method 
which can be used to sample from more general distributions.
According to Greenberg (2008), the accept-reject method can be used to simulate random 
variables from a density function 
( )
when it is possible to simulate values from another 
density 
( )
, and if a number 
can be found such that 
( ) ( )
for all 
. The 
density 
( )
is called the instrumental or candidate density. In order to simulate random 
variables 
from 
( )
Robert and Casella (2010) state that first, we independently 
generate 
( )
and 
( )
. Then, if
( )
( )
we set 
. If not we discard 
. This leads to the accept-reject algorithm 
-
1. Generate 
from 
( )
; 
-
2. Generate 
from 
( ) 
independently of
; 
-
3. Accept
if 
( )
( )
, else reject 
; 
-
4. Return to 1. 
Following Robert and Casella (2010), the cdf of the accepted random variable 
( |
( )
( )
)
is exactly the cdf of 
. That is, 

---

45 
( |
( )
( )
)
(
( )
( ))
(
( )
( ))
∫ ∫
( )
( )
( )
∫ ∫
( )
( )
( )
∫
( )
( )
( )
∫
( )
( )
( )
( |
( )
( )
)
∫ ( )
∫ ( )
∫ ( )
( )
The output is, therefore, exactly distributed from 
( )
.
Considering the efficiency of this method, we note that the probability of accepting a point 
is given by. 
( ) (
( )
( )
) ∫ (∫
) ( )
( )
( )
∫
( )
( )
( )
∫
( )
This implies that we should choose an 
as small as possible in order to maximize the 
probability of acceptance. The algorithm is efficient when 
is as close to 
as possible. 
Maximizing the probability of acceptance is important because as Greenberg (2008, p 67) 
states, “rejected values use computer time without adding to the sample”, therefore, 
deceasing efficiency.