Doctor of philosophy

In summary, Direct Marketers will find the Gini statistic more useful for assessing model performance when the statistical properties of Gini are known and used in the analysis.When comparing two different models, it is possible to determine if one model’s performance falls outside the expected performance range of another model.By utilizing the Gini index and its statistical properties, it is possible to validate a response model with statistical precision.

6.24.9 Model performance

Can be measured in terms of cumulative gains charts referred to a banana charts.

• 
  Confusion matrix
  
• 
  ROC charts
  
• 
  P Gains Charts
  

6.24.9.1 Receiver Operating Characteristic (ROC) Curve

A Receiver Operating Characteristic (ROC) curve is a graphical representation of the trade off between the false negative and false positive rates for every possible cut off. By tradition, the plot shows the false positive rate (1-specificity) on the X axis and the true positive rate (sensitivity or 1 - the false negative rate) on the Y axis.

The accuracy of a test (i.e.the ability of the test to correctly classify cases with a certain condition and cases without the condition) is measured by the area under the ROC curve.An area of 1 represents a perfect test, while an area of 0.5 represents a worthless test.The closer the curve follows the left-hand border and then the top border of the ROC space, the more accurate the test; the true positive rate is high and the false positive rate is low.Statistically, more area under the curve means that it is identifying more true positives while minimizing the number/percent of false positives.



Fig.51: Comparison of R.O.C. curves

Table.48. Confusion Matrix to measure model fit

I/O	NR	R	Sum
NR	A	B	A + B
R	C	D	C + D
Sum	A + C	B + D	Total

The fit of the model can be measured by various measures.In the confusion matrix number of peredictis for response (R) and non-response (NR) for model output (O) and target (T) respectively.

Entry A denotes the number of NR outputs when the target is NR while B denotes the number of R outputs while the target is NR. C denotes the number of NR outputs when the target is R while D denotes the number of R output when the target is R. A model with large A & D is a better one. The accuracy is defined as the ratio of the diagnoal entries A+D over total number.

The performance of a binary classifier with a division thereshold is often plotted using Receiver operating characteristic (ROC) curve.

1 - Sensitivity = C / C+d

1 - Specificity = B /A+B

Using the ROC Curve to Measure Sensitivity & Specificity

Two indices are used to evaluate the accuracy of a test that predicts dichotomous outcomes(e.g. logistic regression);sensitivity and specificity. They describe how well a test discriminates between cases with and without a certain condition

6.24.9.2 Sensitivity

The proportion of true positives or the proportion of cases correctly identified by the test as meeting a certain condition (e.g. in mammography testing, the proportion of patients with cancer who test positive).

6.24.9.3 Specificity

The proportion of true negatives or the proportion of cases correctly identified by the test as not meeting a certain condition (e.g. in mammography testing, the proportion of patients without cancer who test negative).

6.24.9.4 Choosing a Cut-off

The position of the cut-off determines the number of true positives, true negatives, false positives, and false negatives.If the sensitivity is increased (true positives) and more cases with a certain condition can be identified, accuracy is sacrificed on identifying those without the condition (specificity).

6.25 Evaluation criteria for response models

• 
  Percentage correctly classified at the economically optimal cutoff purchase probability (PCC).
  
• 
  Area under the receiver operating characteristics (AUC).
  
• 
  The focus is knowing (1)what the additional predictive power of an additional variable over a model which does not contain this variable over a model which does contain this variable (2) to product purchase behaviour (3) when using correlated predictions the hypothesis testing approach may suffer as multicollinearity may cause inflated variance of the estimate which may in turn lead to insignificant parameters (4) predictive approach to model has gained substantial importance in econometrics at the expense of the hypothesis testing approach, Geisser and Eddy(1979), the increased availability of larger deta sets has played a catalysing effect in this respect because in large samples almost all coefficients become statistically significant, Granyer(1998), and (5) using the predictive evaluation criterion entails the use of separate test sample which follows Morrison’s warning against the upward bias that results from classifying, the same individuals used to calculate the classification model in the estimates of the resubstitution rate.