Comparing real-time, historical, and replay performance

[NinjaTrader_ChelseaB]

Citizens of the NinjaTrader Community,

A common question we hear from clients is 'why are results from backtest different from real-time or from market replay?'.

Live orders are filled on an exchange with a trading partner on an agreed upon price based on market dynamics. Backtest orders are not using these market dynamics. Instead these are filled based on logical rules from processing historical data.

The main reason for large disparities is due to a lack of intra-bar granularity for order fill prices in the backtest, or testing a different data set.

See more about intra-bar granularity.
https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

By default when backtesting no intra-bar granularity is added. An added 1-tick series is necessary to increase fill accuracy to within 1 tick.
Further, TickReplay is necessary for intra-bar indicator values to update and recalculate for on each tick or on price change Calculate setting.

As a heads up, TickReplay cannot be used for accurate order fill prices and High Order Fill Resolution cannot be used in conjunction with TickReplay.


Please note, as a script is enabled live on a chart or on the Strategies tab of the Control Center, the historical data will be processed first before the strategy transitions to processing real-time data. As the historical data is processed, this is the same as performing a backtest and will provide the same historical results as backtesting in the Strategy Analyzer.
https://ninjatrader.com/support/foru...442#post782442


Small differences in fill time, slippage, as well as small differences in when ticks are received and which bar they are included with can cause differences in the fill prices which aggregate in the total performance.

For the tests to be even similar, it is necessary to test the exact same sets of data, without mixing and matching the data types. (Meaning it would not be wise for a comparison of real-time or playback to include historical data and the days tested must start and end on the same session)

Its best to test by starting the strategy and reloading all historical data before running the test in real-time or playback.


Once the basics of getting similar test environments (meaning same data set, and intra-bar granularity) have been satisfied, it becomes clearer how to spot exactly what has caused the differences in the order fills.

Switching to view the chart may show that the charts appear very very similar but there are still differences in the final performance.

Prints can be added to the script (or this information can be written to file if its a large amount of information) to further understand the behavior.

See the post below about using prints for debugging.
http://ninjatrader.com/support/forum...979#post510979


This applies to NinjaTrader 7 as well, except for one large limitation: Tick Replay is a new concept developed for NinjaTrader 8 that is not implemented in NinjaTrader 7.

The major difference with this is that indicators that need to update with CalculateOnBarClose as false will not be able to update with the secondary tick series. (The fill prices will use the secondary series, however, the indicators will not be updating tick for tick and will only update when the bar closes)

Due to this, to get comparable results in NinjaTrader 7 for any scripts that call indicators the script must have CalculateOnBarClose set to True for all environments, real-time, historical (backtest), and replay.


It can also be a challenge to know what information to print, and how to analyze the information once its printed. It was even a challenge for me putting this post together just getting the test data to be the same. (Printing the bar and tick information really helped with this)

Below is a link to a video I've created that demonstrate how to put this all together and analyze a small backtest in all three environments.

https://www.youtube.com/watch?v=2iM5BPpcUBg

Attached is the modified SMACrossOver script I've written to test.
RealtimeReplayHistoricalComparisonsExample_NT8.zip

[simple_goodoboy]

Thank you ChelseaB for this through and timely written response.

I will state my Objective, Problem, Solution, and Questions below as it relates to this subject.

Objective
Confirm to myself that my strategy that looks profitable in back test using CQG free data on 5 min to 1 hour (whatever data series works) data series is showing me near-accurate results.

My Problem or Concern
I have no evidence to trust my back testing results with a degree of accuracy.
I need to know when I click back test or Optimize, there is xx% the results will be accuracy and I have some reliability in my data feed, Strategy Analyzer, and myself to be consistent. I do not care if its 30% accurate, I just want to know or have some idea.

My Solution
For each strategy that shows profitability to my liking in back test simulation do the following:
1. Use Strategy Analyzer to back test the strategy. Choose a series of 300 trades where there are some equally longs and shorts. Document the results.
2. Market Replay the strategy for the same 300 trades. Document the results.
3. Manual Back Test the strategy for 300 trades. Document the results
4. Compare the results of steps 1,2, 3 for accuracy. If decent accuracy, proceed to step 5
5. Run strategy in Sim for 300 trades. Document the results
6. Market Replay for the 300 trades time period is step 5.
7. Run strategy analyzer for the same 300 trades in step 6.
8. Compare step 5, 6, and 7 for accurate. If decent accuracy, proceed to step 9.
9. Go live with cash.
10. As time goes on continue to back test live results with back testing simulation to gain confidence in my data feed.

What do you think of this? lol

QuestionsQuestions

1. Currently I back test with CalculateOnBarClose = true; for all my strategies and indicators. What do I need to do make sure when i test in market replay and sim, CalculateOnBarClose = true; ? See my attachment please

Meaning it would not be wise for a comparison of real-time or playback to include historical data

What do you mean by this statement?

Tick replay becomes necessary for intra-bar indicator values to update and recalculate for each tick or price change.

When using the Strategy Analyzer, do the indicator update on every tick? I believe the answer is No, indicator update on bar close correct only, correct?

However, this cannot be used for the order fill prices, so a secondary series is also necessary for order fill resolution (which cannot be used with Tick Replay).

If my primary data series is 5 min, and secondary series (high fill order resolution set to 1 tick) and CalculateOnBarClose = true, will the strategy trigger every 1 tick or every 5 min when bar close?

This applies to NinjaTrader 7 as well, except for one large limitation: Tick Replay is a new concept developed for NinjaTrader 8 that is not implemented in NinjaTrader 7.

The major difference with this is that indicators that need to update with CalculateOnBarClose as false will not be able to update with the secondary tick series. (The fill prices will use the secondary series, however, the indicators will not be updating tick for tick and will only update when the bar closes)

I am not sure I understand here. What's the purpose of Tick Replay, when in back test tick replay was not used to build the bars for back test?

Due to this, to get comparable results the script must have CalculateOnBarClose set to True for all environments, real-time, backtest, and replay.

I totally agree with this. Whatever showed me profitability in back test, I want the same exact setup for the strategy execution in live, sim, and market replay. I will enable Tick Replay for better results, but CalculateOnBarClose set to True will always be enabled just like it was during back testing simulation.

I apologize for the long post, but I have to get the details somewhat understood and right. Of course there is no future promises with trading, but due diligence is a must for me. I want to understand my platform and everything about it.

[simple_goodoboy]

It can also be a challenge to know what information to print, and how to analyze the information once its printed. It was even a challenge for me putting this post together just getting the test data to be the same. (Printing the bar and tick information really helped with this)

Below is a link to a set of videos I've created that demonstrate how to put this all together and analyze a small backtest in all three environments.

https://www.screencast.com/t/c73xusjul

Attached is the modified SMACrossOver script I've written to test.

Chelsea thank you sooo much for creating those videos and the well in depth analysis.

That was a very good video and explained alot.

Few Questions/Comments
1. I notice you was using Kinetic data feed for your back test and replay analysis. I am using CQG, I assume it does not matter if i want to perform similar test using CQG, correct?

2. Did you have tick replay enabled for the Market Replay and Real Time testing? If so, I am assuming you did because the back test will submit orders on the 1 tick secondary data series. However, because we the back test was on the 1 min primary data series, the strategy was not triggered until the 1 min bar close right? Just trying to understand

3. Please see attachment with questions.

4. Just to makes sure I understand the results. The data was correct, it just the execution time was a bit different from the real time vs back testing, correct? In other words, this due to slippage in real time.

[NinjaTrader_Jim]

Hello simple_goodoboy,

I have no evidence to trust my back testing results with a degree of accuracy.
I need to know when I click back test or Optimize, there is xx% the results will be accuracy and I have some reliability in my data feed, Strategy Analyzer, and myself to be consistent. I do not care if its 30% accurate, I just want to know or have some idea.

As backtesting in the Strategy Analyzer uses historical data, and not real time data or market replay data, it would be possible to have the Strategy Analyzer calculate an "accuracy" metric. There is no data set for comparison Your approach to testing would effectively compare these so you can see how close of a representation your Strategy Analyzer results give.

Meaning it would not be wise for a comparison of real-time or playback to include historical data

...

What do you mean by this statement?

Real Time data and Historical data are different. Historical data is a representation of the market flow and is only meant to give you an idea of the results that the strategy would create. If the strategy processed historical data upon enablement, it would "muddy the waters" as we only want to see what would happen in real time. This is why the Trading Hours template was used to restrict the strategy to only use the data it would process in real time.

When using the Strategy Analyzer, do the indicator update on every tick? I believe the answer is No, indicator update on bar close correct only, correct?

The answer is no, unless you enable Tick Replay for the back test and the calling strategy is set to Calculate.OnEachTick.

If my primary data series is 5 min, and secondary series (high fill order resolution set to 1 tick) and CalculateOnBarClose = true, will the strategy trigger every 1 tick or every 5 min when bar close?

I would advise to test this out to see the behavior. You can create a script that adds the one tick data series, and then you can add a Print() in BarsInProgress == 1 to see if prints are occurring on each tick of the added data series. If you add the data series to the strategy, the strategy will process it in OnBarUpdate().

I am not sure I understand here. What's the purpose of Tick Replay, when in back test tick replay was not used to build the bars for back test?

Tick Replay is meant to recreate historical data with tick data. Its use in the strategy is to allow the indicators to update tick for tick. since the indicator does not add a 1 tick data series for it's own calculations.

1. I notice you was using Kinetic data feed for your back test and replay analysis. I am using CQG, I assume it does not matter if i want to perform similar test using CQG, correct?

This would not matter, as long as you are using the same source of data for comparison.

2. Did you have tick replay enabled for the Market Replay and Real Time testing? If so, I am assuming you did because the back test will submit orders on the 1 tick secondary data series. However, because we the back test was on the 1 min primary data series, the strategy was not triggered until the 1 min bar close right? Just trying to understand

Tick Replay was enabled for the historical back test to get Calculate.OnEachTick to work. As Calculate.OnEachTick can be used freely in Market Replay and Real time data, it would not be needed for this comparison. Furthermore, historical data is restricted by the trading hours template for this comparison when testing with real time and market replay data.

In the test, orders were submitted to the 1 tick data series. The noted difference is the time when the execution happens for historical trades, which is immediate.

Questions in images:

1. The Strategy Analyzer is using Tick Replay to recreate the tick data and use Calculate.OnEachTick.
2. Tick Replay can also be used to call OnBarUpdate() "OnEachTick" or "OnPriceChange" during historical calculations.
3. High Order fill resolution mimics submitting orders to a single tick data series like what is demonstrated here.

Please see the documentation on Developing for Tick Replay and the High Order Fill Resolution.

Developing for Tick Replay - https://ninjatrader.com/support/help...ick_replay.htm

High Order Fill Resolution - https://ninjatrader.com/support/help...FillResolution

Just to makes sure I understand the results. The data was correct, it just the execution time was a bit different from the real time vs back testing, correct? In other words, this due to slippage in real time.

Yes, since the back test fills orders immediately, it would not be show the delay that happens with Real time order fills.

[simple_goodoboy]

Thank you so much Jim for your detailed response.

A few more questions for clarifications:

If the strategy processed historical data upon enablement, it would "muddy the waters" as we only want to see what would happen in real time. This is why the Trading Hours template was used to restrict the strategy to only use the data it would process in real time.

1. How do I prevent my strategy from processing historical data after enabling the strategy in sim mode or real time?

2. In Chelsa attached code, does the following prevent trading on historical data:
if (CurrentBars[0] < BarsRequiredToTrade || CurrentBars[1] < BarsRequiredToTrade || (!Backtest && State != State.Realtime) || (Backtest && (State != State.Historical || !Bars.IsTickReplay)))
return;

3. If I enable the strategy in sim or real mode, how can historical data be used in forward trading to "muddy" the water? I agree, i only want to trade real time data.

The answer is no, unless you enable Tick Replay for the back test and the calling strategy is set to Calculate.OnEachTick.

1. I thought in back test, we can only use Calculate on Bar Close, not every tick?

Tick Replay is meant to recreate historical data with tick data. Its use in the strategy is to allow the indicators to update tick for tick. since the indicator does not add a 1 tick data series for it's own calculations.

1. So if I don't use Tick Replay and I use Calculate.OnBarClose during the back test the indicator/strategy will just calculate their values every every bar close, correct?

I think I understand now. Chelsa wrote the code so the strategy/indicator will calculate on each tick and in order to get historical tick data, we have to enable Tick Replay. Is my understanding correct here?

Also, he added AddDataSeries(BarsPeriodType.Tick, 1); so all orders are placed on the 1 tick chart. This way the back testing for this example will run similar to the market replay and real time testing. Is my understanding correct here?

Thank you kindly. Its slowly sinking in

[NinjaTrader_Jim]

Hello simple_goodoboy,

1. How do I prevent my strategy from processing historical data after enabling the strategy in sim mode or real time?

if(State == State.Historical) return;

2. In Chelsa attached code, does the following prevent trading on historical data:
if (CurrentBars[0] < BarsRequiredToTrade || CurrentBars[1] < BarsRequiredToTrade || (!Backtest && State != State.Realtime) || (Backtest && (State != State.Historical || !Bars.IsTickReplay)))
return;

This part does when we are not backtesting: (!Backtest && State != State.Realtime)

3. If I enable the strategy in sim or real mode, how can historical data be used in forward trading to "muddy" the water? I agree, i only want to trade real time data.

In this particular test, we are comparing Real time data with Historical data. We want that comparison to be pure and comparing Historical results against Real time results that include some Historical data would "muddy" that comparison.

1. I thought in back test, we can only use Calculate on Bar Close, not every tick?

Tick Replay can be used to call OnBarUpdate() "OnEachTick" or "OnPriceChange" during historical calculations. Backtests use historical calculations.

1. So if I don't use Tick Replay and I use Calculate.OnBarClose during the back test the indicator/strategy will just calculate their values every every bar close, correct?

Calculate.OnBarClose will calculate values for the close of each bar. Tick Replay lets you use Calculate.OnEachTick or Calculate.OnPriceChange for historical purposes. You could still enable Tick Replay using Calculate.OnBarClose and use OnMarketData() to utilize the historical tick data.

I think I understand now. Chelsa wrote the code so the strategy/indicator will calculate on each tick and in order to get historical tick data, we have to enable Tick Replay. Is my understanding correct here?

Yep!

Also, he added AddDataSeries(BarsPeriodType.Tick, 1); so all orders are placed on the 1 tick chart. This way the back testing for this example will run similar to the market replay and real time testing. Is my understanding correct here?

Yep! But keep in mind the Order Methods need to use the overload for specifying the BarsInProgress you wish to send the order to.

[NinjaTrader_ChelseaB]

Hey guys,

Jim, thanks for your great answers.

I wanted to further detail that check before the script may begin processing, which checks several things.

if (CurrentBars[0] < BarsRequiredToTrade || CurrentBars[1] < BarsRequiredToTrade || (!Backtest && State != State.Realtime) || (Backtest && (State != State.Historical || !Bars.IsTickReplay)))
return;

The CurrentBars < BarsRequiredToTrade for each series allows the user to prevent trades until their desired amount of bars have passed. This can also help to prevent errors when changing parameters that may cause calls for previous bar information (invalid indexes for bars ago.

The Backtest bool is used so that the user can mark if they are testing with a backtest using historical data, or if they are testing with real-time / market replay data.

(!Backtest && State != State.Realtime)
This indicates if Backtest is unchecked and we are testing with realtime, to stop the script if the script is not in real-time data.

(Backtest && (State != State.Historical || !Bars.IsTickReplay)
This indicates if Backtest is checked, then we need to be processing historical data only and not allowing the script to continue with real-time data. This also requires that TickReplay is enabled so that indicators may update in real-time when Calculate is set to .OnEachTick / .OnPriceChange in historical / backtest.


Also, when you run your strategy live, you probably will want to include historical data as well as historical trades. This always helps with guiding the indicators in the right direction before actual trading starts (which is why a user may want to set BarsRequiredToTrade to a higher number so that indicators are averaging better by the time trading starts).

However, when doing a comparison like this where you are trying to find the very minute details that are causing differences, including that historical data with a real-time test is going to cause very very different calculations and can even cause different start positions.

The backtest is going to start at a specific time and begin testing from there and will not include further historical that would be used to train indicators before trading starts. So for a real-time comparative test, you want the same. Any indicators would need to start building with 0 data and build using purely real-time data (the same as what is happening in backtest).

[simple_goodoboy]

Jim,

Thank you so much for responding to me.

A few more questions please.

Tick Replay can be used to call OnBarUpdate() "OnEachTick" or "OnPriceChange" during historical calculations. Backtests use historical calculations.

1. Does this mean the indicators (ie, emas, mcad, etc) will calculate OnEachTick as well or just the strategy? I get confused when the indicators calculate vs when strategy calculate?
2. By using OnEachTick or OnPriceChange, I assume a back test will take longer and use more computer resources to complete a back test, correct?

You could still enable Tick Replay using Calculate.OnBarClose and use OnMarketData() to utilize the historical tick data.

1. Using Calculate.OnEachTick or Calculate.OnPriceChange for back testing historical purposes is not available in NT7, correct?

2. I plan back test strategies on NT8 using High Fill Order Resolution with primary bar series as 5 min and secondary bar series as 1 tick. I will be using Calculate.OnBarClose.
Do I need to enable Tick Replay or will the 5 min bars get created with OHLC and secondary bar series with 1 tick data?

Thanks just trying to understand.

[simple_goodoboy]

Thank you very much ChelseaB for explaining

A few questions please.

Also, when you run your strategy live, you probably will want to include historical data as well as historical trades. This always helps with guiding the indicators in the right direction before actual trading starts (which is why a user may want to set BarsRequiredToTrade to a higher number so that indicators are averaging better by the time trading starts).

If historical data and historical trades is not included in real time strategy trading, shouldn't the Max bar look back set to 256 bars be enough to train the indicators before taking a trade?

Thanks

[NinjaTrader_Jim]

Hello simple_goodoboy,

1. Does this mean the indicators (ie, emas, mcad, etc) will calculate OnEachTick as well or just the strategy? I get confused when the indicators calculate vs when strategy calculate?

This is discussed in the previously discussed Developing for Tick Replay documentation. If you want to have the added indicator process its historical data through Tick Replay, you can add a reference to the indicator in State.DataLoaded in OnStateChange(), or you can use AddChartIndicator() in a strategy. If Tick Replay is enabled and the indicator is added with one of these methods, the indicator will honor the Calculate setting assigned to it for historical backtests.

2. By using OnEachTick or OnPriceChange, I assume a back test will take longer and use more computer resources to complete a back test, correct?

Yes. More data to calculate will increase the number of resources required for the backtest.

1. Using Calculate.OnEachTick or Calculate.OnPriceChange for back testing historical purposes is not available in NT7, correct?

Correct. Tick Replay allows you to use these Calculate modes in NinjaTrader 8. Tick Replay is not available in NinjaTrader 7.

2. I plan back test strategies on NT8 using High Fill Order Resolution with primary bar series as 5 min and secondary bar series as 1 tick. I will be using Calculate.OnBarClose.
Do I need to enable Tick Replay or will the 5 min bars get created with OHLC and secondary bar series with 1 tick data?

As noted in the Developing for Tick Replay documentation, you cannot enable Tick Replay with a High Order Fill Resolution. High Order Fill Resolution submits orders to a finer data series. You can recreate this behavior by having your order methods submit orders to an added 12 tick data series. If you are using Calculate.OnBarClose, there would not be any benefit to playing back the historical data with Tick Replay.

If historical data and historical trades is not included in real time strategy trading, shouldn't the Max bar look back set to 256 bars be enough to train the indicators before taking a trade?

If your indicators do not require more than 256 bars to look back at a given time for calculations, I would suggest to set the Max bars look back to 256 bars to use less resources.

Please take a moment to review the documentation, notes and implications listed in the Developing for Tick Replay page of the help guide.

Developing for Tick Replay - https://ninjatrader.com/support/help...ick_replay.htm

[simple_goodoboy]

Thank you Jim for responding and the link.

I am understanding now.

If you want to have the added indicator process its historical data through Tick Replay, you can add a reference to the indicator in State.DataLoaded in OnStateChange(), or you can use AddChartIndicator() in a strategy. If Tick Replay is enabled and the indicator is added with one of these methods, the indicator will honor the Calculate setting assigned to it for historical backtests.

So if Tick Replay is not enabled during back test, and only have Calculate.OnBarClose = TRUE on a 5 min data series, then will the indicators calculate every 5 min in back test and in sim/real-time? Sorry if I am confusing you.

[NinjaTrader_Jim]

Hello simple_goodoboy,

I may be misleading you if I answer the question with a simple yes or no. Please consider the following:

Tick Replay is for historical calculations and would not affect real time.

Tick Replay with Calculate.OnBarClose will call OnBarUpdate() for each bar close. (You can use OnMarketData() for the Tick Replay data when using Calculate.OnBarClose.)

Tick Replay with Calculate.OnEachTick or Calculate.OnPriceChange will call OnBarUpdate() with each incoming tick.

Indicators will follow their Calculate setting for historical calculations if Tick Replay is enabled and they have been added appropriately (see help guide) otherwise they will default to Calculate.OnBarClose.

Real time calculations will follow the Calculate setting without restriction.

The answer falls closer to a "yes," but the uses for Tick Replay and how it affects Calculate (for historical calculations only!) should be noted.

[staycool3_a]

I think this should be a sticky or moved to tips/suggestions etc. Probably the single most important post ever.

[Impeesa]

Hello Chelsea

Below is a link to a set of videos I've created that demonstrate how to put this all together and analyze a small backtest in all three environments.

https://www.screencast.com/t/c73xusjul

Link provided requieres FlashPlayer and after December 31, 2020 it can not be downloaded... How can we access to this important contents? Maybe another format file?

Thanks a lot

[NinjaTrader_ChelseaB]

Hello Impeesa,

Open Microsoft Edge browser.

Navigate to the url.

Click the link for https://get.adobe.com/flashplayer/

Click Allow Flash