Same Data, Same Strategy, Different Results?

[aufidius]

I noticed that when moving my strategies over from NT7 to NT8 I got very different results in Strategy Analyzer. So I decided to test the "SampleMaCrossOver" strategy which is the exactly the same in both versions.

So Using Kinetic EOD data I tested the SampleMACross on RB and also got very different results. Anyone have any thoughts on why i'm seeing such a big difference? I'm sure I am missing something here.

NT 8




NT 7

[NinjaTrader_EricB]

Hello aufidius,

Thank you for your post. I tested the SampleMACrossOver in NinjaTrader 8 and I do get the same results that you have.

However, I tested the same NinjaTrader 7 parameters and I do not get the same while backtesting. Can you please test using the SampleMACrossOver attached to my reply?

Please include a screen shot of your results.

[NinjaTrader_Brett]

Thanks for posting,

I checked into further since I expect overall similar results from NT7 to NT8 with standard backtests.

The issue is a difference between the amount of data in NinjaTrader 7 vs NinjaTrader 8.NT7 is only including data from the first expiration date in the database forward which is from 2008. NT8 is returning all data for the entire last contract in the database which is the cause for the different results.

If you change your start date on NT8 to be 7/21/2008 you should see similar results to NT7.

I'm checking into this and look into making a change in NT8 to match NT7 here.

[aufidius]

using your code the results on my machine are the same as before




Interestingly after my post someone privately told me about this article http://www.trade-robots.com/blog/nin...lts-are-random

It seems other people have notice inconsistencies with NT7 in the past.

[aufidius]

Based on your explanation it seems to be that NT8's behavior is the ideal way to work. With NT7 users could be assuming they are testing with 10 years of historical data when they are only getting 7 and they are not told. This could lead to people thinking a system is more robust than it actually is...

Thanks for posting,

I checked into further since I expect overall similar results from NT7 to NT8 with standard backtests.

The issue is a difference between the amount of data in NinjaTrader 7 vs NinjaTrader 8.NT7 is only including data from the first expiration date in the database forward which is from 2008. NT8 is returning all data for the entire last contract in the database which is the cause for the different results.

If you change your start date on NT8 to be 7/21/2008 you should see similar results to NT7.

I'm checking into this and look into making a change in NT8 to match NT7 here.

[koganam]

using your code the results on my machine are the same as before




Interestingly after my post someone privately told me about this article http://www.trade-robots.com/blog/nin...lts-are-random

It seems other people have notice inconsistencies with NT7 in the past.

Both the link that you point to, and your own results are usually an indication that you are not testing the same dataset. For comparison purposes, every single parameter of your tests must be identical. Start dat, end date, time period, possible even the data vendor.

After all parameters are set, you may want to delete your data (or save it off), and download fresh data (to multiple computers, if your tests are not on the same computer).

Have I seen this behavior before? Yes. So far, in every instance, after review, it was something that I did that meant that I was not making identical tests. Every time, I corrected the data and parameters issue, I got identical results in backtest.