Does backtesting work?

[Owlman]

Hi All,

I am in a quandary. I have always believed that backtesting was the only way to fine tune a strategy to be most profitable in the future. Lately (over the last six months or so, and having almost wiped out my account) I have had the opposite experience. When I thought about it I arrived at the conclusion that my belief was based on the assumption that price action next week would be similar to price action last week, and, for quite a long time (over the past couple of years) this assumption seemed to hold up. But now it doesn't seem to any more. Now it seems that fine tuning my strategy based on the last two weeks to maximum profitability produces disastrous losses the following week every time, whereas it used to produce profitable results the following week.

My strategy is a very simple one that follows price action and does not use any indicators. The only parameters that I change are the profit target and stop loss levels (both fixed at a number of ticks) and the size of the trend it looks for. There are no conditions, if it sees my setup it takes the trade. So I don't think I'm curve fitting too much.

I've tried a number of modifications to the strategy over the last few months, but every time I've found the same thing. The strategy can be made to run profitably in back testing, but in real time, disastrous loss.

I'm at my wit's end. Does anybody have any idea why this is happening? Is anybody else experiencing similar results, i.e. backtesting producing radically different results in real trading?

Having said that, I don't expect miracles. I understand that if my backtesting produces a 10% profit then in real trading I would consider a 3% profit a very good result. What I don't get is why I get a 15% loss in real trading, that just doesn't seem to make any sense.

I posted this question on Big Mike Trading too, and got some very interesting and considered responses (thanks to those respondents) but so far I still haven't been able to sort out what I'm doing wrong, if anything, and how I can get my strategy back into profitable territory. All ideas gratefully received!

Ian

[koganam]

You have run into one of the major issues of backtesting. Unfortunately, there is not really enough information here to even begin to see where your particular problem lies.

The first thing to do is to not just backtest, but rather to do a walk-forward test to test the validity of your parameters.

Now, asking some questions about your backtest, are your results in the backtest statistically significant? We cannot know that unless we know how many trades your backtest found, and how the results are distributed. For example, did you find just a few trades with large wins; or many trades, most with small wins or losses and a few trades with very large wins; or many trades with many small wins and a few large losses; or many trades with similar steady wins and losses, but just more wins than losses?

Did you chart your distribution of results to see what kind of curve the results generate? For a good system, you expect a distribution with a small kurtosis and small dispersion.

[NinjaTrader_RyanM1]

Hi Owlman,

Backtesting is a good starting point, but you should definitely do further analysis before running it on a live account. You need a feel for the actual mechanics of your strategy running in real time and can do this on a sim account or with NinjaTrader's market replay. Backtesting is historical only (no guarantee that past = present), lacks granularity, and can be forgiving for the types of conditions that are only present in real time trading.

http://www.ninjatrader.com/support/helpGuides/nt7/discrepancies_real_time_vs_bac.htm

[Owlman]

Thanks Koganam and Ryan,

To address some of Koganam's queries, the strategy did 208 trades in the backtest period (15 Aug to 28 Aug) and blitzed it with a bit of parameter tuning. I use a fixed profit stop and stop loss (different ones for long and short) so the profit or loss on each trade is mostly the same. I also have a condition that the strategy will reverse the trade if a reverse signal is detected, it did this 17 times in the backtest period. It will also unconditionally exit the trade at the end of the trading session. There were 5 of these. (okay, so it has two conditions...)

My stops were set as follows: long profit - 29 ticks, long stop 14 ticks, short profit 21 ticks, short stop 12 ticks. The test was done using a 144 tick chart on the FDAX 09-11 future contract.
Detailed results. In backtest period: 92 winners for 1891 points, 116 losers for loss of 1410 points. In test period (the week following the backtest fortnight): 30 winners for 589.5 points, 57 losers for loss of 706 points. Total profit for backtest fortnight - 481. Total loss for test week - 117 points. Pretty dismal.

I didn't chart the distribution of results, I don't know how to do that. Would I need to calculate a standard deviation in Excel and then graph the result? Needless to say, I also don't know what kurtosis and dispersion are... I found a Kurtosis function in Excel. The Kurtosis of the results (points gained or lost) is -1.7232, but I don't know what the significance of that is... Couldn't find anything about dispersion though.

Cheers,
Ian

[koganam]

Given your description, you have a contrained system where you have 4 possible outcomes (fixed profit/fixed loss on long/short), and one variable outcome (EOD exit) which will, by definition always be less than the fixed values. With a constrained system, dispersion will be fixed, and because there is no continous variation in results, kurtosis actually becomes unimportant.

But let us look at your results:

...
In backtest period: 92 winners for 1891 points, 116 losers for loss of 1410 points. In test period (the week following the backtest fortnight): 30 winners for 589.5 points, 57 losers for loss of 706 points. Total profit for backtest fortnight - 481. Total loss for test week - 117 points. Pretty dismal.

Case 1:
Winning probability = 92/208 = 0.44
Losing probability = 0.56

Case 2:
Winning probability = 30/87 = 0.34
Losing probability = 0.66

In both cases, with information from perfect hindsight available, the strategy shows itself to be a losing one.

It would appear to me that backtesting is telling us quite clearly that if we trade with these parameters, we shall lose. One of the advantages of backtesting is that it is really the only time in trading that we have perfect information on the future. Given any trade, we know exactly what happened after the trade. If given such perfect conditions, the trade method produces losses, it is only reasonable to expect no differently in real time trading.

Your original question seemed to imply that backtesting showed a winning system that went on to lose in real time, whereas the results that you have posted show a losing system, per the backtest. Why then are we discussing whether backtesting works? In this case, it would seem that it most assuredly does.

[lindsayf]

It sounds like to me that you are simply curve fitting your paramaters to a small historical dataset. If you then trade this live with real time data you will get exactly what you have got. Firstly you need to set your paramaters over a large dataset - 1 week is far too small - try 6 months. Then you could run the paramaters over an out of sample dataset for comparison and if results are good then you should, as others have suggested, do a walk forward test ( on SIM) to see how closely the backtest results are matched by real time trading. IF there is an appropriate correlation then you might be onto a profitable method. At this stage you could consider trading live with small position size.

Curve fitting is a big trap indeed.

[Owlman]

Thanks Koganem.

I have to point out though that the system is profitable, if you take into account that the profit target is greater than the stop loss. The total points gain is therefore greater than the total points lost (in the test period), therefore a profitable strategy. I can change the parameters so the profit target is always equal to the stop loss, and my results are quite different.

Thanks lindsayf.

However, I feel I need to point out a couple of things. First off, my strategy is an intraday strategy using a 144 tick chart, most of the trades last less than five minutes. It did 208 trades in two weeks, I think that's a reasonable sample size, and one thing I have noticed is that the further into the past I go, the less the price action in the backtest resembles todays' price action. The point I'm getting to is that I believe that backtesting a strategy like mine on data that is six months old is not going to produce realistic results next week. If you were using a daily chart you would not backtest on data twenty years old. Data that old would simply not be relevant to today's situation, whereas data that is up to two years old would be.

Re the walk forward test, that is precisely what I have done and precisely why I am confused. When I run my strategy using the parameters I've set during the optimisation, on the week following the optimisation period (data the is not included in the optimisation set) the performance is pretty much the exact reverse of what I've achieved in the optimisation period. Trading a small position size is great advice. I trade one contract. You can't get any smaller than that. Well, you can, I could trade zero contracts, then I'd definitely never lose!

Curve fitting is indeed a trap. I experimented with a product called Omnitrader for a while, and it was the curve fitting king. It would run multiple strategies with mutliple indicators in each strategy, and multiple parameters for each indicator, and optimise the lot. Needless to say the performance in backtesting was incredible, but in real time, not so much...

As I mentioned in my original post, my system does not use any indicators and relies only on price action. I can vary the profit and loss targets, and two other parameters related to the strength of the trend it looks for to initiate the signal trigger, but that's it. There is no "if this happens do this otherwise do something else", which is the bane of curve fitters. So I find it difficult to accept that I am excessively curve fitting. There is a fine line of course, but I think there is nothing wrong with optimising parameters to improve performance in backtest. That's why Ninja has the Strategy Analyzer functionality built in, and indeed every trading course I've ever attended has emphasised the importance of backtesting to optimise trading strategies. So when does optimising become curve fitting? Tough call, not sure I know the answer...

Cheers,
Ian

[koganam]

Thanks Koganem.

I have to point out though that the system is profitable, if you take into account that the profit target is greater than the stop loss. The total points gain is therefore greater than the total points lost (in the test period), therefore a profitable strategy. I can change the parameters so the profit target is always equal to the stop loss, and my results are quite different.

...

Cheers,
Ian

The problem with designing any system that relies ab initio on the relative sizes of targets and stops to compensate for a greater probability of losing is that the system then loses tolerance. Essentially, we would say that the system is unstable, because a relative small variation on the winning/losing probabilities is immediately magnified. Unfortunately, it is difficult enough coming up with a system that consistently has more winners than losers, so starting off with a system that is the other way round just makes things that much harder.

Consider this. Once a trade is placed, what is the probability that it will make money? Not that your target will be hit. Just that at any time, the chance that that trade will be in profit or loss. Paradoxically, the answer has nothing to do with when, why or how the trade was initiated. There is only one answer: the price will either be above the entry price, or below it, so that probability is 50%. That is why so often stops get hit before the market turns on a dime and makes a beeline for the erstwhile target. Some refer to this as the "perversity of markets", but it is just statistics. In fact, many researchers (Connor's Group is one), have shown that adding stops to ANY strategy reduces the profitability in backtest. Does that mean that we should trade without stops? I never would, because I do not have infinite funds.

Let us take another mathematical look at your system.

Assuming that the backtest results will carry forward into the future, and unrealistically assuming zero trade costs, what is the expected return of your system? As a result of the 50% rule above, We shall just use your overall probabilities, for either long or short.

Expectation = probability * return, so

For your long trades:

Win expectation = 0.44 * 29 = 12.76
Loss expectation = 0.56 * 14 = 7.84
Overall expectation = (int)(12.76 - 7.84) = 4 ticks per trade

For your short trades:

Win expectation = 0.44 * 21 = 9.24
Loss expectation = 0.56 * 12 = 6.72

Overall expectation = (int)(9.24 - 6.72) = 2 ticks per trade

We could average that and say that you have a system that would overall return maybe 3 ticks per trade.

Your actual overall return was slightly lower at 481/208, which as an int value is 2 ticks.

However, to show how sensitive your system is to variation, let us use the most optimistic valuation and say we have an expectation of 4 ticks, implying that we are using our best target/loss ratio of 29/14, a smidgen over 2:1.

The question then is, at what ratio of losing to winning trades, does our expectation reach zero? (At zero returns, the trade costs will make sure that we go broke).

29 * winExp - 14 * (1 - winExp) = 0

Which resolves to a win expectancy of 0.32.

So, at its most optimistic results, with your system moving from 56 losses per hundred trades to 68 losses per hundred trades, a mere 12 percentage points per hundred, the system goes into losses overall. Does an 12 % variance in trade performance look very big? Not to me, it does not. What about if we used the most pessimistic lowest return of 2 ticks (from short trades), or even the mean return of 3? I will leave that as an exercise.