Bayesian Logistic Regression Models for Credit Scoring by Gregg Webster

 
2.3 Markov Chain Monte Carlo Methods
Conjugate priors for the logistic regression model do not exist which makes sampling from 
the posterior distribution difficult. Gelfand and Smith (1990) introduced the turning point 
for the use of MCMC methods in statistics. These MCMC methods are methods which are 
used to obtain samples from a posterior distribution when it is not analytically possible to 
obtain the posterior. MCMC methods were introduced by statistical physicists in the 
1950s. Metropolis 
et al
. (1953) introduced an algorithm known as the Metropolis 
algorithm. The algorithm was then generalized by Hastings (1970) and became the 
Metropolis-Hastings (MH) algorithm. The algorithm works by constructing a Markov 
chain which has a stationary distribution equal to the target (posterior) distribution. This is 
achieved through a kind of accept-reject strategy. A value is proposed and this value is 
accepted or rejected according to a rule which ensures that the Markov chain generated has 
a stationary distribution equal to the target (posterior) distribution. It resulted in renewed 
interest in Bayesian statistics through the use of modern computers being able to perform 
algorithms - such as the Gibbs sampler and Metropolis-Hastings (MH) algorithm. 
Determining integrals is of vital importance in obtaining the posterior distribution. The 
Metropolis-Hastings algorithm is more general than the Gibbs sampler. The MH algorithm 
is the principle algorithm on which Bayesian logistic regression is based. The MH 
algorithm adopts a kind of accept-reject strategy to the simulation while the Gibbs sampler 
is a special case, which can be used when it is possible to sample from conditional 
distributions. Most studies which consider Bayesian logistic regression use the MH 
algorithm to sample from the posterior (Ziemba, 2005; Wilhelmsen 
et al
., 2009). This is 
because the Gibbs sampler cannot be used directly as one cannot sample easily from the 

---

18 
conditional distributions. Holmes and Held (2006), however, demonstrated how inference 
can be done efficiently in the Bayesian logistic regression model using a Gibbs sampler. 
They showed that the conditional likelihood of the regression coefficients is multivariate 
normal when certain auxiliary variables are introduced. This, then, allows for efficient 
simulation using a block Gibbs sampler. Simulation of the posterior distribution is thus 
either done using the Gibbs sampler or the MH algorithm. The MH algorithm is, however, 
far more popular with Bayesian logistic regression as the model is not complicated by 
additional auxiliary variables.