Backtesting vs real-time results

[Billy Valentine]

When backtesting a strategy, does the Strategy Analyzer calculate every tick on a bar identical to real-time trading? Specifically, are the entries and exits triggered in a backtest identical to entries and exits had the strategy been traded forward in real time for the same period as the backtest?

[NinjaTrader_JoshP]

Nope there is no inside the bar logic. Please check out this article: http://www.ninjatrader-support.com/H...sBacktest.html

[unclelou]

Backtest exits

Josh,
So if I understand corrrectly, backtest exits will show up in the P&L summaries at the bar close price, even if the strategy exit occurred mid-bar? Ex. T1 = +10 ticks, but the bar closes +25 ticks. The backtest results/execution page will show T1 at +25?

If so, this makes it tough to trust the backtest results. It seems to overstate profits?

Thanks in advance for your reply.

[NinjaTrader_JoshP]

Exits happen intrabar if they should occur. It is entries that do not happen intrabar. Reason exits are allowed is because you touch a price and generally you can be filled there. Reason entries are not allowed is because the bar you are using as your signal bar is already closed therefore untradeable.

[unclelou]

That's good to know and will work fine for me. However, when I compare the backtest results for yesterday on the Strategy Analyzer "Executions" tab, the T1's that are listed do not jive with what actually happened with the strategy. Ex: Trade #1 hit T1 @ +8 and was filled at +6 (slippage). But the SA Executions tab show that T1 was hit at +15. This discrepancy with T1 occurred with all three of the profitable trades that day (the exits at the stops seem to work fine for the losers). Am I doing something wrong?

[NinjaTrader_JoshP]

Are you citing discrepancies between live trading and SA backtesting? Backtesting and live trading will rarely ever yield the same results and are not directly comparable.

[unclelou]

Discrepancies between live trading using an autotrade strategy with Ninja vs. backtesting that same strategy.

[NinjaTrader_JoshP]

Yes. Then you will always have discrepancies as the ones outlined here: http://www.ninjatrader-support.com/H...sBacktest.html

[unclelou]

If "backtesting and live trading will rarely ever yield the same results and are not directly comparable," it would follow that backtesting will not help me determine if a strategy has been successful in the past and will not help me optimize a strategy going forward? So, with all due respect, what good is Ninja's backtesting, really? How does it help traders? Am I missing something?

Thanks.

[NinjaTrader_JoshP]

unclelou,

This is the nature of backtesting in general. It is analogous to the issue in statistics for fitting the data to the curve versus fitting the curve to the data.

This is why many traders and system developers push for not only backtesting, but also extensive forward testing. What backtesting provides you with is a general idea of how your strategy can behave. It is not information to be solely relied on for decisions.

There is also a thing known as Walk Forward Optimization which essentially takes a period of time and backtests parameters against that period. Then it takes those settings and walks forward to see how it does against a new time period. Even this is just a shot in the ballpark, but it is a shot that provides information as to a possibility.

There is no certainty in trading and as such backtesting, optimizing, walk forward optimizing, and forward testing all will never produce you with information on true results. All of them provide a view into a possibility for what your strategy could produce. You take and analyze all the information to get a range of behaviors. Then it is up to you if decide this range is acceptable for live trading or not.

[Stainless2000]

I am having similiar problems with trailing stops and backtesting. It appears to me if the bar is greater in size than your trailing stop, you are stopped out immediately even though in real time the market was going in your direction within that bar.

[NinjaTrader_JoshP]

Stainless2000,

Please review the article about discrepancies between real-time and backtesting: http://www.ninjatrader-support.com/H...sBacktest.html

With only OHLC data there is NO way to know how your stop would have behaved in real-time. You say the bar was going in your direction. You cannot quantify that with just OHLC data. Anything could have happened within the dynamics of that bar. The bar could have been going up, trail stop pushes up, then price could have came down, triggered stop, then price goes back up. You just don't know.

[Stainless2000]

All true. But the consistancy of the action and its timestamp lead me to logical conclusions. It is simply impossible for the same thing to happen everytime. I grant you that half the time or even somewhat more than that the trade might look the way it does, and that the bar would not be going my way, but not everytime.

[Stainless2000]

Ok, I read the article and I think I see what my problem is and why trailing stops don't work well in backtesting. Since the test is only run on Bar Close if the stop is smaller than ANY move in that direction within the bar regardless of if it was before or after the trade itself you are stopped out. This is why the stops report having been hit instantly, becuase they were actually calculated simutaneously...Is this correct?

[NinjaTrader_JoshP]

Stainless2000,

There is nothing to be changed here. It is the best of two evils. You can either have our conservative approach or just randomly assume you were lucky and have runners that overstate your startegy's profit. Again, we are arguing about the nature and limitations of backtesting. It is what it is. This is why backtesting should never be relied on for a definitive performance profile of your strategy.