Evaluating the Quality of a Trading System
There are two aspects in assessing the quality of a trading strategy: historical performance and probability of future performance. Most measures do a very good job in calculating historical performance numbers, but are not very helpful in predicting future behavior of a trading strategy. My most prominent example is max. drawdown: Easy to calculate, but rather useless for prediction (see below).
We designed a measure called “SysQ” (System Quality) which helps to estimate the future performance of a trading strategy. In short SysQ means:
Expected Yearly Profit with a Drawdown of 10%.
Or, the longer, somewhat more precise version: If we adjust position sizes (or leverage or exposure) to arrive at an expected P95 drawdown of 10%, then the expected annual return is the SysQ (System Quality).”
Design Goals
We had the following design goals in mind when we developed SysQ:
- Should be a practical, useful number which relates to an experienced property of the trading system
- Instead of describing the exact properties of the historical (past) equitiy curve it should say something about the future prospects of the trading system.
- Should return the same number after position sizes / leverage are changed
- Should be independent of time frame or length of equity curve
- Should work for all types of trading systems (Daily, intraday) all asset classes (stocks, Commodities, Currencies, etc) and even price series.
- Should be a stable measure: small changes in Portfolio, Basket or system parameters should result in small changes of SysQ.
Existing Performance Measures
In a review of existing performance measures it became obvious, that most measures don’t cope very well with return distributions other than the normal distribution, with serial correlation of returns and other non-linearities.
The Problem
The least common starting point in evaluating the performance of indices, benchmarks, portfolios or trading strategies is the resulting equity curve. The segments of an equity curve of a trading strategy is the combination of several random walks (while one or more positions are open) or a constant (if there is no open position). Mathematically one considers the daily returns of this equity curve. But the volatility varies fast, values are not normally distributed, there are many outliers (fat tails). This makes it very hard to describe the properties of such an equity curve analytically. Even the “arithmetic average” isn’t well defined in such a case. To summarize, the equity curve shows quite some problematic properties:
- not enough data. Usually we work with daily return values. But even with several years of backtest/simulation data this is still a small set for a reliable prediction of future behavior.
- values not normally distributed, frequent outliers
- serial correlation. If there are crashes, all open positions tend to move in the same direction, for several days. Usually this is what creates the max. Drawdown. If this serial correlation isn’t considered correctly, we get bad estimates for future drawdowns.
- highly variable variance
It is nearly impossible to describe an equity curve accurately with a few numbers or a working model, which is the usual way to create predictions for future behaviour.
Fortunately, the solution for all these problems and questions has a name: The Bootstrap, invented by B. Efron in 1979.”
The Solution: Expected Max Drawdown
We use the bootstrap to generate a large number of synthetic equity curves and measure the max. drawdown of each. The drawdown which is not exceeded in P% of all trials is the “Expected Max. Drawdown at Percentile P” or eMDDP.
The key to this procedure is to use three day segments of returns to construct the synthetic equity curve. With this three days segment a critical part of the serial correlation is captured and makes the synthetic equity curve much more realistic.
The eMDDP is a very useful measure in itself and can be used to construct various useful statistics:
- eMDD95: Expected Drawdown, not excceeded in 95% of all cases.
- System Quality: 10 * APR / eMDD95