Bayesian Logistic Regression Models for Credit Scoring by Gregg Webster

 

---

23 
3.2 Bayesian Statistics 
 
3.2.1 Bayesian inference 
Bayesian inference provides a useful way to combine expert knowledge (prior belief) with 
data to arrive at some posterior belief. All Bayesian inference is conducted through the use 
of Bayes’ theorem (Press, 1989; Bernardo and Smith, 2000; Lee, 2004; Greenberg, 2008; 
Ntzoufras, 2009). 
Press (1989) explains that when one has a prior belief (called a prior distribution) before 
one observes the data, Bayes’ theorem gives a mathematical procedure for updating the 
prior belief to arrive at a posterior distribution. The derivation of Bayes’ theorem makes 
use of conditional probabilities, 
( | ) ( ) ( )
and
( | ) ( ) ( )
. 
Therefore, 
( ) ( | ) ( ) ( | ) ( )
which leads to Bayes’ theorem: 
( | ) ( | ) ( ) ( )
.
(3.1) 
 
3.2.2 Prior density, likelihood and posterior density functions 
Following Greenberg (2008) and setting 
(a parameter or vector of parameters) and 
, we have the following for continuous or general
.
( | ) ( | ) ( ) ( )
(3.2) 
where
( ) ∫ ( | ) ( )
. Equation (3.2) is the basis of Bayesian statistics and 
econometrics. We now analyse Equation (3.2) in detail. 
( | )
, the left-hand-side of 
Equation (3.2) is the posterior density function of 
θ 
|
 y
. 
( | )
is the density function of 
the observed data 
when the parameter value is 
. 
( | )
is called the likelihood function 
and is a function of 
θ
once the data are known. 
( )
is called the prior density and 

---

24 
represents beliefs about the distribution of 
before seeing the data 
. These beliefs can 
come from the researcher’s knowledge or from other external sources. The prior 
distribution usually depends on parameters called hyperparameters. 
( )
normalizes the 
posterior distribution so that integrating Equation (3.2) with respect to 
θ
yields 1. Equation 
(3.2) can also be written as 
( | ) ( | ) ( ) 
(3.3) 
The right-hand-side of Equation (3.3) does not integrate to 1 but it has the same shape as 
( | ) 
The posterior distribution contains all the information we have about 
.