Journal of Risk Model Validation

Steve Satchell
Trinity College, University of Cambridge

In preparation for this issue of The Journal of Risk Model Validation, we were assured by people whose opinion we value that model risk is, in principle, not quantifiable in a practical or consistent way – that we would receive good papers on isolated problems but find no coherent principles. This view has a strong weight of evidence and experience behind it.

The problem of model risk quantification starts early: even the attribution of loss from a broken model is hard to define and harder to measure. It is complicated further by the fact that model risk is at once an operational risk and a specification risk. How can we find a unifying principle for such different phenomena? And finally, quantified model risk struggles to find an aggregatable and business-meaningful scale of measurement.

It is therefore a pleasure to have found four papers for this issue that equally and firmly oppose this view and address the challenges positively. Before going into specifics, let us outline their common approach and their consistent message.

• To make progress, focus on model specification risk: the risk of loss caused by a poor or incorrect model choice. This makes the problem tractable because the residual analysis of model operational risk has its own principles and approaches.
• Model risk is a relative risk. Model risk measurements are best made relative to a baseline model, or from among a group or continuum of candidate models, and the relative variation among the models is the range of model specification risk.
• Where a continuum of models is to be tested, relative entropy measures of model variation are most natural in theory as well as being practically implementable.
• Specification risk variation can in practice be traced to actual business impact and losses. It is not just about abstract model variation.

The impact of specification risk is drawn out clearly in the issue’s first paper: “Model risk in mortality-linked contingent claims pricing” by Gareth W. Peters, Hongxuan Yan and Jennifer Chan. Here, the three components of a longevity model – mortality, interest rates and the interaction between them – each have structural options, and the specification of the model is an important risk. To explore the variations that are possible, the authors select from three mortality model structures, including their own recently developed multivariate long-memory model, while also varying in four possible ways the causal interactions of the mortality model with a Vasicek interest rate model (they note that other interest rate models could also be tried). This gives 12 test models, the spread of whose parameters and business outputs, such as annuity prices, quantify the model specification risk by inspection. The novel long-memory modeling and underlying Bayesian approach are of separate interest as well.

In a different business context, a similar approach to specification risk is explored in “Quantifying model selection risk in macroeconomic sensitivity models” by Joseph L. Breeden and Nikolay Dobrinov, our second paper. This concerns the stress testing of credit default for banking loan portfolios, where the macroeconomic timeseries models that underly stressed probability of default (SPD) use factors whose selection and preprocessing (lags and differences) are open to choice. The many possible variants are filtered automatically according to basic performance criteria and then ranked by the Akaike information criterion, giving a list of the top 31 models (with a further expert assisted ensemble model added for anchoring). These are all models that are close to good enough to be selected, and their variation around the optimum choice is a gauge of model selection error. For each of three different portfolios examined, and each of several stress test scenarios, the resulting spread of estimates of SPD is a credible quantification of the variation due to specification risk.

Stress testing also motivates the third paper in this special issue: “Quantification of model risk with an application to probability of default estimation and stress testing for a large corporate portfolio” by Michael Jacobs Jr. Here, a structured model combining time-varying hazards and macroeconomic factors is used to estimate stress impacts on a large corporate portfolio. The specification risk is found by exploring not a finite set of test models, but a continuum of variations around the baseline model. The paper explores a small Kullback–Leibler neighborhood of the baseline model and compares that with another model optimized by the minimum relative entropy principle. The difference between the baseline and the optimum gauges the specification error of the baseline. By repeating this analysis with different baseline model assumptions, the author determines the specification error under different baseline assumptions in actual stress testing outputs over time under different stress scenarios.

The fourth and final paper in the issue takes a similar approach, has a similar aim and also uses relative entropy. However, the context in “Model risk quantification based on relative entropy” by Daniel Arrieta is quite different: the specification risk of a portfolio investment strategy in an incomplete market, where arbitrage-free strategies are not unique and in which variation materially affects the derivative security pricing. The choice of optimal strategy is like model specification, and the risk of this choice is naturally quantified by entropy relative to a baseline strategy. By varying the baseline assumptions, the specification risk of each strategy can be assessed, and the author illustrates how the risks of certain assumptions (eg, about implied volatilities) are quantified under this method using Monte Carlo methods/