Backtest and Optimizer inconsistencies between NT7 and NT8

[guillembm]

I have been testing how optimizer and backtest work with my strategies and I have been comparing the results between NT7 and NT8.

I found that there are important inconsistecies between them and I would like to acknowledge if I am doing something wrong. To check if the problem really came from my code I made the tests with Sample MA crossover strategy that comes with both NT7 and NT8 and that are supposed to give the same results.

As you can see in the screenshots I attach this strategy with the same parameters and same backtest settings give completely different results.

Data has been obtained from IB using NT7 and has been migrated to NT8 during a fresh install.

Can you please check the screenshots attached to see if there is any explanation or error?

[NinjaTrader_BrandonH]

Hello guillembm,

Thank you for your post.

It is likely that NinjaTrader 7 is evaluating more historical data than NinjaTrader 8. This could be caused by NinjaTrader 8 not having as much saved historical data as NinjaTrader 7.

Please try running a backtest for the SampleMACrossOver strategy on 1 year of data (ex: start date: 11/01/2019 end date: 11/01/2020) for NinjaTrader 7 and NinjaTrader 8. Are the results similar between the two platforms?

Also, please check the Trades view to see if the trades are starting at the same time.

Thanks in advance, I look forward to your reply.

[guillembm]

It is true that the shorter is the analyzed period the higher are the similarities obtained between NT7 and NT8. This happens when backtesting and also when optimizing.

The saved data in NT8 is the same than NT7 because it has been fully migrated. I took a look at the data folders to confirm this.

I really don't know how to cope with this, specilly when I optimize. Does it mean that NT7 results are more reliable than NT8 or the opposite?

Is there any way to increase the similarity of the results?

[NT-Roland]

Hi guillembm,
Something must be different. Look at trade duration in bars. Completely different.
Would you mind to post the both trade lists again sorted in ascending order? No need to include MFE, MAE, ETD. However, it would be most useful to actually see the exact date and time (not truncated). This will help to spot which trades were taken/missed in both NT versions.
NT-Roland

[guillembm]

Sorry, but my previous message was incomplete.

The difference is that in NT8 "Exit on close" was checked.

But still, the behaviour I explained happens. I verified that this happens after unchecking "Exit on close".

[NT-Roland]

Hi guillembm,
Exit on session close Yes/No can already have a big impact. You also might want to check, if NT7 also uses "Break at EoD" which you seem to have activated in NT8.
NT-Roland

[guillembm]

"Break at EoD" is always checked. I read in some posts that NT7 breaks the bars by default at EOD.

To see an example of what happens when the analyzed periods are different I attach two examples. As you can see, when periods are longer differences become more relevant.

You are porbably thinking that I am too strict but these differences become significant when the strategies are complex.

At this point I am wondering what results to trust. I am thinking in getting the results from NT8 and backtest them in NT7 to see if the difference is acceptable.

[NT-Roland]

Hi guillembm,
If that was my strategy, I'd definitely want to understand the exact reason for the difference.
Although cumbersome, I recommend to check the trades one by one. As soon as you spot a difference, check the chart.
Is the chart data exactly the same and you still get a different trade or do you already have a difference in your charts?
NT-Roland

[guillembm]

Thanks NT-Roland,

I managed to have acceptable differences between NT7 and NT8 playing with the number of GO generations. If I backtest the results obtained in NT7 I get almost the same. I think differences are tolerable.

I will carry on with this

[NinjaTrader_BrandonH]

Hello guillembm,

Thank you for your note.

Please run a backtest in NinjaTrader 7 and NinjaTrader 8 for the SampleMACrossover strategy using the GM instrument and historical data for the dates 11/01/2019 - 11/01/2020. Do you receive the same results for both NinjaTrader 7 and NinjaTrader 8? Please see the attached screenshot of the test we ran on our end showing the same results in both NinjaTrader 7 and NinjaTrader 8.

Steps to reproduce:

NinjaTrader 7:

• Download historical data for the GM instrument with a start date of 11/01/2019 and end date of 11/01/2020 using the Historical Data Manager. (Tools > Historical Data Manager > Load tab)
• After downloading that historical data, open a Strategy Analyzer window (New > Strategy Analyzer)
• In the Strategy Analyzer window right-click on the GM instrument and select Backtest.
• Enter the Backtesting parameters seen in the attached screenshot and click the 'Run backtest' button.

NinjaTrader 8:

• Download historical data for the GM instrument with a start date of 11/01/2019 and end date of 11/01/2020 using the Historical Data window. (Tools > Historical Data > Download tab)
• After downloading that historical data, open a Strategy Analyzer window (File > New > Strategy Analyzer)
• In the Strategy Analyzer window, set the Instrument field to use the GM instrument.
• Enter the Backtesting parameters seen in the attached screenshot and click the 'Run' button.

Compare the two Strategy Analyzer reports and report back with your results.

I look forward to your reply.

[guillembm]

This is correct. Resulting data is the same if we do not consider some irrelevant differences.

I am wondering what is the problem. Are there errors when data is migrated from NT7 during a fresh install? Does it happen always or only with IB data? Can I rely on migrated data or I have to download all the data again?

My strategies need several years of data to work. Loading it again is a big pain.

[NinjaTrader_BrandonH]

Hello guillembm,

Thank you for that information.

There is not a problem with the data migrated from NinjaTrader 7 to NinjaTrader 8.

Similar results between NinjaTrader 7 and NinjaTrader 8 should be expected, but exact results require the exact same data and the same order fill processing as well as the exact same strategy logic.

NinjaTrader 7 and NinjaTrader 8 use different historical data servers. Tick data on NinjaTrader 8's historical data servers has sub-second granularity while NinjaTrader 7's historical data servers do not. As the data is recorded differently, it would not be possible to get results to line up exactly.

Please let us know if you have further questions.

[guillembm]

Are you referring to IB servers or Kinetick?

[NinjaTrader_BrandonH]

Hello guillembm,

Thank you for your note.

I was referring to Kinetick in my previous post but I see you are using Interactive Brokers.

NinjaTrader 7 and NinjaTrader 8 have different historical fill algorithms which would cause different backtesting results to be seen. Limit orders in particular are handled differently between the two platforms. You could see some variances with how limit orders are handled if you test other strategies. However, you should be able to achieve the same consistency as long as Trading Hours, Break at EOD and other settings are minded in addition to using verbatim data. Please ensure your PC is set to the same time zone that is configured in NinjaTrader 8, and please also make sure both platforms have MergePolicy set to Merge back adjusted.

Merge Policy can be set in NinjaTrader 7 under Tools > Options > Data, and can be set in NinjaTrader 8 under Tools > Options > Market Data.

Please let us know if you have further questions.