Bayesian Logistic Regression Models for Credit Scoring by Gregg Webster

3.3 Generalized Linear Models 
 
3.3.1 Introduction 
Generalized linear models (GLMs) were introduced by Nelder and Wedderburn (1972). 
These models are an extension to the normal linear regression models and are based on the 
exponential family of distributions. A GLM has the basic structure 
( 
)
where 
( 
)
, 
is a smooth monotonic “link function”, 
is the 
i
th row of a model matrix, 
, and 
is a vector of unknown parameters. Also, 
belongs to some exponential family 
distribution. The exponential family includes many distributions such as the Poisson, 
Binomial, Gamma, Normal and Inverse Gaussian distributions. A distribution belongs to 
the exponential family of distributions if its probability density function has the form 
( )
( )
( )
( ) 
(3.7) 
where 
are arbitrary functions, 
is
an arbitrary dispersion parameter which 
represents the scale, and 
is known as the canonical parameter, which represents location.
The expectation and variance of 
are now derived. The log-likelihood of 
given a 
particular 
is
( ) ( )
( )
( )
( )
. 
Differentiating with respect to 
gives 
( )
( )
. Therefore, 
(
)
( )
( )
( )
.

---

28 
Using the result that 
(
)
, the expectation of 
is 
( )
( )
.
(3.8) 
Now, finding the second derivative with respect to 
gives
( )
( )
Using the general result
(
) (
)
, we have 
( )
( )
(
( )
( )
)
Hence
( )
( )
(
( ))
( ( ))
, which leads to the variance for 
( )
( ) ( )
. 
(3.9) 
If 
is known, there is no difficulty working with GLMs using any function of
( ) 
If, 
however, 
is unknown, it is common practice to assume 
( )
, where 
w
is a 
known constant. Hence, 
( )
( )
(3.10) 
Since the binomial distribution will be used in this study, it is important to show that the 
binomial distribution is a member of the exponential family. The probability mass function 
of a binomial distribution is
( ) (
)
( )
for 
, where 
is the probability of success.
We have 
( ) (
)
( )
(
(
)
( )
)
( (
) ( ) ( ) ( )) 
( (
) ( ) ( ) ( )) 
( (
) ( ) ( ) ( )) 

---

29 
( (
) (
) ( )) 
[
(
) ( )
(
)]
(3.11) 
Comparing Equation (3.7) to Equation (3.11), we see that 
(
)
, 
( ) ( ) ( )
, and 
( ) (
)
. Therefore, the 
binomial distribution is a member of the exponential family. 
(
)
is the canonical 
link function and is called the logit link. The canonical link is mathematically and 
computationally convenient. However, other choices may also be used. The parameters of 
a GLM can be estimated using maximum likelihood and an iterative procedure called 
Iteratively Re-weighted Least Squares (IRWLS).