Journal of Risk Model Validation

Steve Satchell
Trinity College, University of Cambridge

This issue of The Journal of Risk Model Validation contains three interesting papers: two that suggest that conventional model fundamentals can be improved upon as tools of forecasting; and one that discusses how model risk management in financial institutions could be adapted to reporting academic research.

In the issue’s first paper, “Lessons for academic research from model risk management in financial institutions”, Mahmood Alaghmandan and Olga Streltchenko discuss aspects of model risk management in financial institutions that could be adopted to improve the process of conducting (and publishing) academic research. The authors claim that these aspects could identify and mitigate existing limitations, decrease the possibility of erroneous results and prevent fraudulent activities. As a lifelong academic, I find such ambitions very interesting. The authors are quite right to address such problems: research management is usually dreadfully executed by academics. Alaghmandan and Streltchenko present three core principles of financial risk management and propose their application in academia: model ownership, documentation and effective challenge. They explore the concept of model ownership and the pivotal role of the model owner in financial institutions, proposing its equivalent in academia. Subsequently, they discuss the use of documentation in financial institutions and then elaborate on the need for a similar use and scope of documentation in academia. Finally, they explore various aspects of effective challenge in model risk management. They argue that establishing a framework rooted in these principles in academia would improve the quality of academic research and reduce errors in research projects. In some extreme cases, it could control for academic misconduct, consequently reducing the substantial financial costs incurred by academic institutions. I am much encouraged by this paper and, indeed, by all papers that use risk model validation in a broader context than credit risk, as much of what we do should be applicable to a wide range of problems.

Our second paper, “Dissecting initial margin forecasts: models, limitations and backtesting”, is by Vladimir Chorniy and Sergii Arkhypov. They state that initial margin (IM) has a significant impact on counterparty exposure, pricing, capital and limits. This makes the modelling of future IM a critical element of counterparty risk management and pricing. They claim that current risk management approaches, which they review in their paper, consider IM as synonymous with value-at-risk (VaR) and thus use methods such as forward VaR for forecasting, while some approaches have proposed backtesting of VaR forecasts for model verification. The authors make the case that these approaches are limited and that they have also biased the industry regarding the use of backtesting and model verification. Their paper attempts to correct this bias. First, they highlight that IM is not VaR but merely its approximation. Then they show that even with an “IM is VaR” assumption, the forecast of IM is a forecast of a forecast, which is principally different from “just” forward VaR forecasting. They demonstrate the fundamental limitations that follow from this assumption, and after reviewing the literature on IM forecasting they propose a generic backtesting and verification framework that accommodates both forecasting limitations and existing models. For model verification they consider two approaches: direct backtesting/monitoring and an “elicitability”-related approach. Their analysis also includes the special case of a bank’s model of IM/VaR being a near-perfect replica of an exchange’s IM. There are many places in finance where we are, or should be, interested in forecasts of forecasts. One example that springs to mind is forecasting the future bitcoin price via an examination of market participants’ views on future market levels; indeed, in any market that is weak on fundamentals, assessing beliefs is about the only weapon we have at our disposal. This paper provides a structure for thinking about such problems.

The third and final paper in this issue, “Incorporating financial reports and deep learning for financial distress prediction: empirical evidence from Chinese listed companies” by Jiaming Liu, Ming Jia, Yanan Hao and Lu Wang, is a comparative study on text information processing methods for financial distress prediction. Two methods are used to convert financial reports into vectors: word2vec and bidirectional encoder representations from transformers (BERT). Two other methods, weighted word2vec and BERT-sentence, are also utilized for enhanced text processing and report quantification. Experimental results based on a data set of 62 312 Chinese listed companies from 2000 to 2021 show that weighted word2vec achieves an average prediction accuracy of 85.27% in cross-validation and 84.67% in sliding-time window validation. Based on their findings, Liu et al argue that incorporating semantic information from management discussion and analysis (MD&A) can significantly improve the performance of distress prediction models for listed companies, regardless of the text processing technique used. This is a strong claim to make. They also claim that text-based features become comparable to financial indicators and even surpass them as the prediction horizon extends. Combination features offer greater enhancement than financial indicators, especially for longer prediction horizons. The authors offer a comprehensive validation of MD&A for the purpose of predicting financial distress, and they firmly believe that it serves as a valuable tool in mitigating risk within financial risk management. It would be most interesting to develop ideas about why text might be better than some financial indicators; this may suggest that market sentiment rules markets more than fundamentals. I hope you find this an interesting and thought-provoking paper.