The effect of imperfect data on default prediction validation tests

Heather Russell, Douglas Dwyer and Qing Kang Tang

Abstract

ABSTRACT

Analysts often find themselves working with less than perfect development and/or validation samples, and data issues typically affect the interpretation of default prediction validation tests. Discriminatory power and calibration of default probabilities are two key aspects of validating default probability models. This paper considers how data issues affect three important power tests: the accuracy ratio, the Kolmogorov-Smirnov test and the conditional information entropy ratio. The effect of data issues upon the Hosmer-Lemeshow test, a default probability calibration test, is also considered. A simulation approach is employed that allows the impact of data issues on model performance, when the exact nature of the data issue is known, to be assessed. We obtain several results from the tests of discriminatory power. For example, we find that random missing defaults have little impact on model power, while false defaults have a large impact on power. As with other common level calibration test statistics, the Hosmer-Lemeshow test statistic simply indicates to what degree the level calibration passes or fails. We find that the presence of any data issue tends to cause this test to fail, and, thus, we introduce additional statistics to describe how realized default probabilities differ from those expected. In particular, we introduce statistics to compare overall default probability level with the realized default rate, and to compare the sensitivity of the default rate to changes in the predicted default probability.