Strategy Analysis - Analyzer vs. Playback drastic differences

[Jim H.]

I have a very basic strategy that I have been testing out. When I run it through the strategy analyzer in Backtest (or optimization) mode, the results come out very high (a net profit), but when I run the exact same strategy through the replay, using the same timeframe, the results are extremely different. I could understand a few ticks here or there, but why would they be so drastically different?

Shouldn't they be almost dead-on identical?

[NinjaTrader_Kate]

Hello Jim H.,

Thank you for your post.

You should expect that a strategy running real-time (live brokerage account, live market simulation, in Playback using Market Replay etc...) will produce different results than the performance results generated during a backtest. This difference may be more easily seen on certain Bars types (e.g. Point and Figure, Renko) than others due to their inherent nature in bar formation.

During a backtest you can select conservative or liberal fill algorithms which will produce different results. Fills are determined based on 4 data points, OHLC of a bar since that is the only information that is known during a backtest and there will be no intra-bar data. This means actions cannot happen intra-bar, fills cannot happen intra-bar. All prices and actions come from and occur when the bar closes as this is all the information that is known.

Because of this, OnBarUpdate will only update 'On bar close' as it does not have the intra-bar information necessary for 'On price change' or 'On each tick'.

Also, here is a link to the differences on real-time vs backtest (historical).
http://ninjatrader.com/support/helpG...ime_vs_bac.htm

Adding intra-bar granularity can help with this.

Intra-bar granularity adds a second data series such as a 1 tick series so that the strategy has finer granularity in the historical data in between the OHLC of the primary series. This allows for more accurate trades by supplying the correct price at the correct time for the order to fill with.

In NinjaTrader 8, there have been two new enhancements so that programmers do not have to manually add this secondary series and code the script to for high accuracy fills (Order Fill Resolution) and for intra-bar actions (TickReplay).

Here is a link to our forums that goes into depth on using Order Fill Resolution and Tick Replay to ensure your backtests are as close to real time results as possible:

https://ninjatrader.com/support/foru...mance?t=102504

High Fill Order Resolution and TickReplay cannot be used together. If it is necessary to have both, it is still possible to add intra-bar granularity to a script in the code itself for order fill accuracy and use TickReplay to update indicators with Calculate set to OnPriceChange or OnEachTick historically.

Please let us know if we may be of further assistance to you.