Appendix

Terry Benzschawel

A1 RECEIVER OPERATING CHARACTERISTIC AND CUMULATIVE ACCURACY PROFILE CURVES

The receiver operating characteristics (ROCs) and cumulative accuracy profiles (CAPs) are closely related. Both are used to evaluate models’ abilities to classify events into different categories. The model requires that distributions of the classifier’s responses to each type of signal at various intensities be known.

Model example

For this example, assume the model is designed to predict whether or not corporations will default on their debt in the next year. To construct the distributions, all scores output by the model are ranked. Then, beginning with the riskiest (ie, those with highest default scores) and moving down the predicted probability of default (PD) axis, we can plot relative frequency distributions for each of the two classes, “defaults” and “non-defaults”. The resulting distributions are plotted in Figure A.1.

Figure A.1(b) shows the ROC. This is the ratio of “hits” (ie, the number of defaulters above the default threshold, as a function of “false alarms”, firms above the threshold that did not default. The filled circles in part (a) of Figure A.1 are points on the non-default and default