Journal of Risk Model Validation

As usual, this issue of The Journal of Risk Model Validation consists of four papers: two papers on stress testing and two regional analyses. We are delighted to include an analysis of the BRICS countries (Brazil, Russia, India, China, South Africa) and also a case study of a Polish bank.

In our first paper, "Risk model validation for BRICS countries: a value-at-risk, expected shortfall and extreme value theory approach" by Jean Paul Chung Wing and Preethee Nunkoo Gonpot, the authors employ value-at-risk (VaR) and expected shortfall (ES) as risk measures to assess the competency of several volatility models based on the stock index of the BRICS countries since the financial crisis. The aim is to determine the most appropriate model for each country as well as identify a set of common models for the BRICS countries. Some models are ruled out under the presented backtesting operations, and the models that are not rejected are ranked using two loss functions. In spite of returns displaying large kurtosis, the models with normal innovations generally prove better at giving estimates of VaR and ES. For example, GARCH and FIGARCH capture Brazilian and Russian returns. GJR-GARCH-N is the best model for South Africa, which is the only country that rejects the stationary extreme value theory (EVT) approach. Despite these contrasting results, the authors find that it is possible to obtain more than one model that can be used to model all the BRICS countries: the IGARCH-N AND EGARCH-EVT approaches.

The issue's second paper, "Loss given default modeling: an application to data from a Polish bank" by Marek Karwa´nski, Michał Gostkowski and Piotr Jałowiecki, discusses the fact that Basel II allows banks to determine capital requirements using an internal-ratings-based (IRB) approach. Under the IRB approach, one of the key parameters in the regulatory capital formula is loss given default (LGD). This paper compares two methods of estimating LGD: a beta regression model and a multinomial logit (MNL) model. The authors' calculations were conducted for overdrafts of small and medium-sized enterprises using data provided by a Polish bank. The results indicate that the MNL model is better for modeling LGD than the beta regression model.

The third paper, "Stress testing and model validation: application of the Bayesian approach to a credit risk portfolio" by Michael Jacobs Jr., Ahmet K. Karagozoglu and Frank J. Sensenbrenner, is concerned with regulatory issues since the global financial crisis. In particular, there is currently interest in stress testing, partly due to the impact of model risk, and implemented supervisory requirements in both the revised Basel framework and in the Comprehensive Capital Analysis and Review (CCAR) program. The authors contribute to the literature by developing a Bayesian based credit risk stress testing methodology that can be implemented by small to medium-sized banks, as well as presenting empirical results using data from recent CCAR implementations. Through the application of a Bayesian model, the authors can formally incorporate exogenous scenarios and also quantify the uncertainty in model output that results from stochastic model inputs. They contribute to the model validation literature by comparing the proportional model risk buffer measure of the severely adverse cumulative nine-quarter loss estimate, which is a common way to estimate the statistical uncertainty generated by a model. They compare the Bayesian model with the frequentist model and find the Bayesian to be 40% higher in measuring uncertainty. As for the model validation exercise, the Bayesian model outperforms the frequentist model statistically significantly according to the cumulative percentage error metric, by 2% (respectively, 1.5%) over the entire sample period (respectively, the downturn period). My only comment here is that a Bayesian model is only as good as the priors employed.

Our fourth paper, "Comprehensive Capital Analysis and Review stress tests: is regression the only tool for loss projection?" by Pawel Siarka and Lina Chan, presents a cross-sectional stress test analysis of major US banks, focusing on wholesale commercial and industrial loans in the context of the Comprehensive Capital Analysis and Review promulgated by the US Federal Reserve Board. They model gross charge-off rates for selected banks in the United States and demonstrate how the composition of the bank portfolios affects overall losses. They compare the results of the autoregressive
moving-average model with exogenous inputs with those of the modified one-factor model, and, moreover, they present an alternative method that uses an EVT approach to compare the results of a traditional direct macro-sensitive model with the results of a model leveraging latent market factors. The results of the comparative analyses show that their proposed method gives the banks an alternative perspective on potential losses under the stress test scenarios and that asset correlations are different between banks. Such differences lead to various gross charge-off dispersions. In fact, the borrower asset correlation increases the unexpected loss. The authors also show that the expected value of maximum losses in the alternative EVT approach in a given horizon corresponds closely to the severely adverse scenarios prescribed by the US Federal Reserve Board.

Looking through this issue's papers, it appears to me that much of the analysis is comparative validation. That is to say, researchers look at a variety of different models and select the one that is best in some predefined sense. This is opposed to considering a specific model and looking at perturbations from it. We can therefore think of validation as being analogous to the discussions of hypothesis testing in which statisticians compare general to specific methodology versus specific to general methodologies.

I would also like to make you aware that we have sent out a survey asking for your feedback on Risk Journals. You can access the survey at: www.risk.net/journal/feedback. Your participation is appreciated.

Steve Satchell
Trinity College, University of Cambridge