Differences between backtesting and real

[lju45]

When I backtest with the strategy analyzer with market replay activated I get different results than when I test the strategy with the data in real time using the strategy in real. Why does this happen? I attach two images so you can see the differences.
The image with two operations that were successful are in bactesting with the strategy analyzer but with the real data, it did three operations instead of two and only one of them was successful.

[NinjaTrader_Jim]

Hello lju45,

Thanks for your post.

Market Replay simulates realtime data processing and backtesting uses historical processing. Both are different.

Also note that the Strategy Analyzer should not be used when connected to Playback as the Playback engine will effect what data is available in the Strategy Analyzer.

Discrepancies between Live, Realtime, and Backtesting

It is important to understand the differences between each form of testing your strategy as some have more true-to-market results than the other, but at the cost of more intense computation. It is possible to get these to be close, but getting matching results may be too ambitious of an expectation. I'll briefly describe the differences and link more detail and then provide some direction for getting more accurate results in backtesting.

Order Fill Simulations will be different between realtime data simulations and historical simulations. Market Replay (NT7) or Playback (NT8) connections will simulate realtime data, so in the case for Replay/Playback and real live data, bid/ask and volume data will be used to simulate the fills. Historical data only uses the OHLC of the data series that the order is submitted to simulate the order fill and there is no actual intra bar movement with processing the order fill. Adding a single tick data series to have your strategy submit orders to can help to get more accurate order fills.

Additionally, historical processing will follow Calculate.OnBarClose processing unless Tick Replay is used. (Note: Tick Replay allows for intrabar actions, but is not intended for intrabar order fills.)

Live trading and order fills on paper trading accounts will have the exchange fill the order from their actual market dynamics.

More information on the differences between historical and realtime order fills can be found below.

Discrepancies between realtime and backtest — https://ninjatrader.com/support/help...ime_vs_bac.htm

To add a single tick data series to your script for more accurate backtest results, please see the documentation below.

Backtesting with intrabar granularity— https://ninjatrader.com/support/help...ipt_strate.htm

For a more comprehensive study on these differences, my colleague Chelsea has a series of videos showing how these tests can be made as tight as possible.

Comparing real-time, historical, and replay performance — https://ninjatrader.com/support/foru...mance?t=102504

We look forward to assisting.

[Apo84]

Hi,

I have a problem, the difference between historical data and real-time simulation (or real market).

The program (TEST1.txt):
- Candle: 1 minute;
- Enter Long in the trade at the beginning of the candle at market value (EnterLong function);
- Exit Long in the trade at the beginning of the new candle at market value (ExitLong function);

In backtesting the program is perfect. In real-time simulation and in the real market I have the problem. I find acceptable a delay in entry and exit time (a minite and a few seconds), and that the entry/exit values are not the same. The problem is that profit is always worse in simulation than in historical data.
Why?

I tried to insert MultiTime 1 second (TEST2.text) but the situation it's the same.

It's not possible than the profit is always worse.

Thanks Alberto

[cincai]

This is because you didn't include the solution to solve the real life data.

[Apo84]

cincai what does it mean? Alberto

[NinjaTrader_Jim]

Hello Apo84,

The materials linked in post #2 will be very important to understand the differences between historical processing and realtime processing. These are fundamentally different and you will always get different results. I highly recommend reviewing these in depth for full understanding. Let us know if you have questions on those materials.

Historical processing uses the data series that we submit orders to to fill orders while realtime processing has all intrabar movement and can use active market data to fill orders. Since a data series is a snapshot of OHLC points of the data series that we use to estimate where orders fill, performance cannot be 1:1 with realtime data.

Submitting orders to a single tick data series can make order fills as accurate as possible when filling orders and this is going to be more accurate than submitting orders to a 1 second data series.

I also see that your strategy is using Calculate.OnEachTick. Historical processing defaults to Caclulate.OnBarClose behaviors unless Tick Replay is used. This explains why the market orders are signaled at the end of bar.

Using Tick Replay - https://ninjatrader.com/support/help...ick_replay.htm

We look forward to assisting.

[Apo84]

Hi,

thanks for the answer.

I think I understand the difference between realtime and historical simulation and also the use of MultiTime. I read post # 2 and analyzed the tick values with Matlab. But the question is the same.

The profit is always worse than the theoretical one (obtained with the analysis of historical data and with Matlab calculations). Why?
I understand that the profit cannot be the same as that obtained on historical data, but why is the profit obtained on real operations (or on simulated operations in real-time) on over 1000 tadres is lower than predicted?

Exemple: LONG

Historical data Simulation Real-time
Enter: 12000 Enter: 12001.25
Exit: 11985 Exit: 11984.75
Profit: -15.00 Profit: -16.50
So for 90% of the trades

I hope it's clear, I apologize for my not perfect English.

Thanks Alberto

[NinjaTrader_Jim]

Hello Apo84,

You have to look at what trades are generated, where the entry signals are made, and where fill prices are.

All of this will be different when backtesting a strategy that uses Calculate.OnEachTick/OnPriceChange since historical processing will default to OnBarClose. When you would be live testing the strategy, intra bar orders will be calculated, and those orders will be filled intrabar. In a backtest, those orders will be calculated following Calculate.OnBarClose behavior, and the orders will fill based on the data series you submit the order to.

If you combine using Tick Replay with submitting orders to a single tick data series, you can get as close as possible to results generated with Market Replay in the Playback Connection. Playback/Sim101 will still be more accurate since bid/ask/volume is used, and not just the price values of Last ticks. (Using Tick Replay allows intra bar OnBarUpdate calculations, while submitting orders to the single tick data series allows them to be filled with the most accuracy we can achieve in a backtest.)

Please let us know if you have any additional questions.

[Apo84]

Thanks for the answer. I have another question.
Is there an article explaining the difference between Real-Market and real-time simulation? Maybe with examples and numbers.
Thanks Alberto

[NinjaTrader_Jim]

Hello Apo84,

This would be described in the link below.

Discrepancies between realtime and backtest — https://ninjatrader.com/support/help...ime_vs_bac.htm

Live order submissions reach the broke'rs order routing servers and an actual exchange so there is added latency between realtime simulations and live order fills. This adds slippage since it takes time for the order to reach it's destination and get filled. Orders submitted to the exchange will be filled according to the actual order book as well. (This slippage will be variable and could not be quantified as a general metric. Traders that are farther away from the exchange will likely have higher latency and slippage than someone who is closer.) The order's position in the order book will also be considered when filling the order. I.E. if someone got their order in before you, their order will fill first.