Backtesting of a probability of default model in the point-in-time–through-the-cycle context

Mark Rubtsov

Abstract

This paper presents a backtesting framework for a probability of default (PD) model, assuming that the latter is calibrated to both point-in-time (PIT) and through-the-cycle (TTC) levels. We claim that the backtesting scope includes both calibration testing, to establish the unbiasedness of the PIT PD estimator, and measuring calibration accuracy, which, according to our definition, reflects the magnitude of the PD estimation error. We argue that model correctness is equivalent to unbiasedness, while accuracy, being a measure of estimation efficiency, determines model acceptability. We explain how the PIT-based test results may be used to draw conclusions about the associated TTC PDs. We discover that unbiasedness in the PIT–TTC context can take three different forms, and show how the popular binomial and chi-squared tests focus on its strictest form, which does not allow for estimation errors. We offer alternative tests and confirm their usefulness in Monte Carlo simulations. Further, we argue that accuracy is tightly connected to the ranking ability of the underlying rating function and that these two properties can be characterized by a single measure. After considering today’s measures of risk differentiation, which claim to describe the ranking ability, we dismiss them and conclude that they are unfit for purpose. We then propose a modification of one traditional risk differentiation measure, namely, the area under the receiving-operator-characteristic curve (AUC), that makes the result a measure of calibration accuracy, and hence also of the ranking ability.