Questioning the numbers

Model risk has become a key focus for financial institutions. But how do dealers ensure their models are implemented correctly and that they accurately reflect the risks an institution is running? By Laurence Neville

The dislocation in the structured credit market last April and May caused a widespread rethink of activities by both investors and issuers. It also resulted in a re-evaluation of the use of models to price, manage and understand risk, as they were perceived to have failed to explain some of the market movements of that time. The re-evaluation of models has since acquired momentum outside the credit market.

Today, the consensus opinion about what happened last year in the structured credit market is that the models should be cleared of most of the blame – what occurred was a result of market dynamics rather than model risk. As Stephane Delacote, London-based head of structured credit trading and arbitrage at BNP Paribas, notes, the crisis was caused by too many people making the same trade and rushing for the exits at the same time.

Nevertheless, this experience has informed current thinking on the role and value of models in pricing and managing risk. "People have inevitably become more cautious in how they apply models," says Delacote. "If you followed the model during the correlation dislocation earlier this year, you would have bought protection at the top of the market. No-one should be trading just based on the model."

At BNP Paribas, the merger last March of option trading, which tends to be focused on market behaviour, and correlation trading, which is traditionally a quant-type business, means that there is an equal understanding of the importance of models and markets in explaining price behaviour, says Delacote. "The mix of cultures reflects the way markets behave in different ways at different times," he explains.

A similar conclusion has been reached at Bear Stearns, where model validation is integrated with product risk management. "For model-dependent product areas, we've always had a heavy dose of quantitative people in our product area team," says Robert Neff, global head of risk management at the firm, in New York. "We think it's important to have a meshing of those two units rather than thinking about models in isolation."

While the theory of models remains crucial, the practicalities of implementation demand an understanding of positions, markets and overall risk profiles in order to avoid model risk, which is generally defined as the risk that the wrong models are used, that they are implemented incorrectly, or that the models fail to accurately assess the risks an institution runs in its positions.

Increased risk

Typically, model risk becomes greater over time – the further from the trade date, the more likely the output of a model will deviate from observed market prices, explains Neff. "Typically, model risk is low on day one, as many products are transacted in competition and therefore valuation of the trade is easier to validate. Model risk comes in later as the trade begins to drift away from the execution level," he says. "Subsequently, you have model dependency for valuation and hedging parameters as time goes on. If you're lucky, the market might evolve in a way that the model risk declines for a given product, but that cannot be relied upon."

Market pricing dominates how Bear Stearns considers its risks. "We operate in a mark-to-market framework, so the primary function of the model is to answer the question: what is the subsequent valuation of a position according to the market," says Neff. "We use models to come up with prices, and we use the same models to value those positions in the time after that and to help us formulate strategies for managing risk."

Bear Stearns' use of models to value positions for mark-to-market purposes highlights a broader trend in the use of models. Where once predictive judgements would have been made about the value of an asset or liability through a model based on fundamentals or assumptions about market changes, models are now generally used in a different way: to explain current market pricing.

"When traders make prices, they are now much less guided by models than five or 10 years ago, when models had a more prescriptive role," says Riccardo Rebonato, global head of market risk and global head of the quantitative research team at Royal Bank of Scotland in London. Instead, supply and demand have become the dominant factor in explaining prices, he says. "The model is largely just there to explain yesterday's move."

Part of this is down to accountancy changes such as IAS 39 – banks now have to mark-to-market, as opposed to marking-to-model. "Therefore, the price that you observe in a transaction – even if it is irrational or incompatible with the model – is the price," says Rebonato.

Consequently, more emphasis is placed on calibrating models to fit current prices. "All the emphasis is now put on recovering, as closely as possible, today's price information, regardless of whether the model turns out to be rather implausible," says Rebonato. "Models have become more interpolators among and extrapolators beyond observed prices rather than predictive tools to gauge where the price 'should' be."

Rebonato points to the interest rates market, where the explanation of the behaviour of interest rate swaps given by Nasser Saber, an adjunct professor at New York University and general partner of Saber Partnership, a trading and risk management firm, gives a snapshot of prices across maturities. "This does not mean that models are useless," Rebonato says. "In normal market practices, they do tell you how to hedge and neutralise normal market moves. However, supply and demand has become king."

The problem with using models to reflect observed pricing rather than construct valuations based on fundamentals is that not all pricing can be explained by theoretical constructs. As seen during the events in the structured credit tranche market, markets can behave in ways that models would not suggest because of extraneous events or supply and demand factors.

But mark-to-market is steadily transforming the derivatives market into something more akin to a market in the underlying than to a market in a 'redundant' security, says Rebonato. "You don't need a model to tell you what the price of a Treasury will be, and if you want to trade a Treasury you do so based on your views of macro and supply and demand issues. It is a stretch to say that is analogous to the derivatives market. But derivatives are becoming more like that," he says.

The debate about models is still largely ignoring the point that using models to explain observed prices is problematic, says Stephen Blyth, head of European arbitrage trading at Deutsche Bank in London. "People still aren't asking the right questions," he says. "They are still asking, 'Which model should I choose?' as if there is one correct model. But there is no right model."

This is because marking-to-market for regulatory purposes – the most rapidly growing use of models – has subtle differences from using models for risk management. "Attempting to hedge to market is fraught with difficulties," says Blyth. "The market can be viewed as an exogenous force that reconciles supply and demand and no model can be relied upon to capture that dynamic."

Blyth points to the liquid swaptions market to illustrate his point. "You can begin to hedge the interest rate exposure of a receiver swaption because models and the market under virtually all conditions will agree that as rates go down the value of the receiver goes up, and so the hedge is to pay on a swap," he explains. "But as the market rallies, what, for example, happens to volatility? Under different market supply and demand dynamics, the change in volatility as rates move can be different, and thus the amount of delta required to hedge is different. Does one continually readjust the model according to these supply and demand changes? In this case, one can no longer hope to replicate the price given by the model in the first place."

Traders need the appropriate quantitative framework to make decisions, but models are just tools to aid decision-making, notes Blyth. "There is no model that would have told you how to hedge against market behaviour such as that which occurred in the structured credit market last year or in the Bermudan interest rate swaption markets five years ago." Market prices may similarly not recognise long-term value in a portfolio – but a trader may be able to.

Blyth believes that there will always be judgement involved in the use of models. "It is far better to make a fundamental judgement of the value of things – using models as an aid to the decision process – rather than rely on a market price that just reflects the current, often short-term, supply and demand dynamic. When you see a price, even in a liquid market, you need to ask yourself, does it make sense? The market price should be just the start of how you consider value or risk."

Guarding against model risk

Given the financial implications for institutions that inaccurately use models, it is not surprising that most have rigorous procedures for verifying their systems. Bear Stearns, like most financial institutions, uses just one model to calculate its official position – for marking its books to market and generating its risk exposure – but has numerous ways of ensuring that the model is giving an accurate picture. One method used by Bear Stearns to check the accuracy of the model is the integration of product teams responsible for mark-to-market validation and understanding market dynamics with model validation teams, which focus on the theoretical implications of models. The intention is to create a greater overall understanding of the variety of inputs to pricing behaviour. Teamwork and additional analysis is especially important in preventing model risk in situations where an institution is active only on one side of a market, says Robert Neff, global head of risk management at Bear Stearns in New York. Another way of ensuring the accuracy of a model is to stress test it, according to Neff. "Stress testing has a high correlation with model risk because with most modelled products it is clear what scenarios you could be exposed to," he says. "So the risk assessment comes from comparing how your book is positioned in various scenarios with what your model is telling you." While Bear Stearns and other financial institutions might use one model for their regulatory calculations of risk and marking-to-market, many other models are used to verify accuracy. "We exchange values for collateral purposes with our dealers, and if a dealer is using another model then we have an ability to compare values over time," says Neff. "Should there be divergence, it would be flagged." Similarly, alternative models are run by quantitative teams for comparative purposes and by traders to allow them to check prices. "We are constantly looking at alternative models," says Neff. "If one of our developers has a model on his PC that would suit our requirements better, then I want to know about it and so do our traders."

Risk