It’s complicated: why backtesting is so challenging, but so important

Hiroshi Tanase, executive director of product analysis and design at S&P Global Market Intelligence, explores why, with the implementation of phase five of uncleared margin rules last September and with the phase six roll-out just around the corner, firms have been looking more seriously at backtesting and trying to make sense of it

Hiroshi Tanase, S&P Global Market Intelligence

Does backtesting apply to you?

Whether a phase five or six firm must conduct backtesting varies by jurisdiction. US regulators, for instance, don’t require all firms to backtest, while those in some Asian geographies do. In Europe, most regulators apply the principle of proportionality, giving smaller firms less onerous backtesting requirements than their larger counterparts. Backtesting rules in many other jurisdictions are still evolving, with regulators yet to define how to apply them.

Regulatory requirements aside, backtesting is a crucial part of the International Swaps and Derivatives Association’s standard initial margin model (Simm), serving as the validation linchpin for margin calculations, and is a key element of sound risk management and good governance. But it can also be one of the most complex and challenging aspects of the process.

Unpacking the complexity

When considering backtesting, it should first be defined which type is being discussed: dynamic or static. Dynamic backtesting (also called dynamic monitoring) compares Simm calculations with the portfolio’s actual profit and loss (P&L) over a 10-day margin period of risk. In effect, these comparisons match the calculated Simm against the real-world experience on a daily basis, enabling a firm to monitor Simm’s adequacy in real time – especially during periods of increased market volatility. While the computations required for dynamic backtesting are less onerous than those for static backtesting, firms must take great care to ensure consistency between the portfolio underlying the Simm and the P&L calculation. This is important because of trade events that may take place during the 10-day period.

Static backtesting, meanwhile, is a less frequent but far more involved process. It uses a quarter-end snapshot of portfolio information and Simm. It then calculates and applies hypothetical P&Ls using the historical movements of the relevant risk factors, typically consisting of the most recent three years plus one year from a severe stress period, such as the global financial crisis that began in 2007–08. Several significant challenges emerge from this requirement:

1. Sourcing historical data. While the Covid‑19 pandemic produced a few months of stress for markets in the first half of 2020, the financial crisis more than a decade prior remains the most recent sustained stress period. As a result, performing this type of backtesting requires access to historical data from more than a decade ago, which not all Simm calculation providers can access easily.

2. Creating theoretical trading data. Some of the equities underlying derivatives portfolios may not have existed during the stress period. Thousands of companies and securities have listed initial public offerings in the years since the global financial crisis, and so have no trading data during a sustained period of stress. In those cases, firms must create theoretical trading performance data (in effect, estimate how such securities would have performed if they had traded during the period) to complete their backtesting calculations.

3. Calculating over 1,000 scenarios. With around 250 trading days per year, the three-plus-one time period requires firms to compare roughly 1,000 scenarios against their portfolios and gauge how many of those scenarios deliver a P&L value above the calculated Simm.

Backtesting as a best practice

Backtesting is more than a regulatory hurdle to clear. It also serves an essential purpose for risk management strategies.

On the one hand, backtesting offers the quantitative detail needed to determine whether Simm is sufficient. If it isn’t, backtesting results provide the proof and justification to call for additional margin from counterparties to cover the shortfall.

On the other, backtesting also provides a check on counterparties’ calls for additional margin. If a counterparty finds that the Simm is insufficient, it may demand additional margin. Without backtesting yourself, the only option is to take the counterparty’s word for it and pay up. However, armed with your own backtesting data, you may find no shortfall exists or that the shortfall is not as significant as the counterparty claims.

In short, without conducting backtesting yourself, you are at risk of providing too much margin to counterparties or asking too little margin from them.

Finding the right partner

Despite these benefits, firms may still avoid backtesting unless required because it is such a laborious process. Many phase five and six firms don’t have the internal resources to backtest with the speed, accuracy and precision the process requires.

Working with an external service provider can ease much of the burden, though not all margin calculation vendors can handle the full scope of backtesting. Given the level of precision and complexity involved, phase five and six firms should check to ensure their margin calculation provider has the proper expertise and tools. Key areas to check include:

• Rich historical data to backtest to the three-plus-one requirement
• Robust processes and methodologies to create proxy data for securities that did not trade for the entire backtest period
• A proven track record of successful backtesting results with other clients.

The right service provider can address pain points related to backtesting and provide tools to easily integrate the process into existing platforms. Such capabilities can enable smaller firms to ensure their Simm margins are conservative enough for even the most stressful real-world scenarios without getting bogged down in the complexities of the process. 

 

Initial margin – Special report 2022
Read more

Sponsored content