Different stratergy backtest results on two computers

[tdouglas]

Hi,

I've recently installed Ninjatrader on a second computer and ran a back and restore to copy over all my settings, strategies etc... from the original computer to the new computer. After this I used Strategy Analyser on the new computer to back test and it produced very different results from back testing the same strategy on the original computer. On both computers I ensured that I was using the same data ranges etc...

I read some of the forum posts and proceeded to reset the database on both computers and they now produce the same results.

This concerns me though as I'd assume after configuring Ninjatrader on a second computer it would download the same data as was used on the first computer?

In case it's of use Esignal is my data provider, before resetting the databases the results were very different - on computer a $4000 net profit $8000 gross, computer b $2000 net profit $25000 gross so I'd assume this couldn't be to do with any data fixes?

Thanks
Tim

[NinjaTrader_ChelseaB]

Hi tdouglas,

Thanks for your note.

There are few things in the database that can have an effect on your backtests:

Instrument rollover dates - the rollover dates are updated between versions of NinjaTrader and any time instruments are reset or the database is reset.

Session Templates - if any of the exchanges have updated their hours NinjaTrader will create a new session template which will be updated when you upgrade, reset instruments, or reset the database.

Instrument Settings - this wouldn't be an automatic change, but if you have modified any instruments in the Instrument Manager this would have an affect.

Also when backtesting you will want to ensure the interval is the same and that the end date does not include today. (Because today's data will be different as new data comes in)

Also, you will want to ensure that all data is reloaded from the server and not saved from a live chart. Meaning open the instrument in a chart and select Reload All Historical Data. Different timestamps can cause different results. Re-downloading from the server will ensure the time stamps are the same.

For me to know for sure what is causing the difference, I would need your database file from the old computer and new computer while the issue still exists. Unfortunately, it may be too late for this.

Please let me know if you encounter the issue again so that I may inspect your environment.

[tdouglas]

Thanks for the detailed response.

In the back test there are 500 instruments, all from the London Stock Exchange (in which the exchange times haven't changed although would the change to British summer time cause creation of a new template?).

None of the instrument settings have been changed (although I initially change the codes they used for Esignal but this was before the backtest on the first computer).

The time interval was the same.

A handful of the instruments (maybe 10) had been viewed on charts but out of the 500 I wouldn't expect that to of had such an extreme effect.

Unfortunately you're right in that I don't have an earlier copy of the database.

Would I be correct in thinking that that the safest way to ensure that the back test is as accurate as possible would be to reset the database frequently?

Also a related question.... when connected to my data provider (Esignal) the back test takes significantly longer compared to when being disconnect (up to 5 mins compared to 20 seconds). I thought that as the back test had previously ran Ninjatrader would have all the required data (providing the data ranges and type were the same) so wouldn't need to pull more data from Esignal?

Thanks for your time.

Tim

[NinjaTrader_ChelseaB]

Hi Tim,

When backtesting the script disconnected vs connected do you get the same results?

Backtests will try and download historical data if it finds any missing. This can often happen everytime if the end date is set to today's date or yesterday's date (as some sessions are open overnight).

If you set the end date to a date before yesterday, and run the test twice, does the script take just as long to backtest connected as when disconnected?

[tdouglas]

I've ran four tests in the order below

1) Disconnected, completed in 5 seconds

2) Connected, completed in 5 minutes and had significantly different results to test 1

3) Disconnected, ran in 5 seconds again and had the same results as test two (so I'm assuming there was missing data from test 1)

4) Connected, what surprised me here though is that it took 5 minutes again and the results were slightly different to tests 2 and 3 (and very different to test 1)

All tests were ran within a 20 minute windows.

In all tests date ranges were 01/01/2010 to 10/06/2013. To ensure it's nothing to do with the strategy itself the tests I've ran for this post are using the SampleMACrossOver strategy that comes with Ninjatrader.

I've attached a spreadsheet containing the results. The backtest is on 500 instruments on the London Stock Exchange using Esignal as the data provider.

Thanks
Tim

[NinjaTrader_ChelseaB]

Hi Tim,

By using the date 10/06/2013 as the end date you will always be including today's data. If there is any new data at all, all data will be re-downloaded from the historical data server for your test.

Please set your end date to 6/21/2013 and run this test twice again, both times while connected. Please let me know if the second test still takes 5 minutes and has different results.

[tdouglas]

Hi,

Thanks but Ninjatrader is using UK date format DD/MM/YYYY (I assume it's picked up from the host computer as I don't remember setting it anywhere). When I enter 6/21/2013 I get an error message saying "Property value is not valid", clicking on details shows "06/21/2013 is not a valid value for DateTime".

This evening I've ran further tests with various different end dates, all results were the same whether connected or disconnected but the connected tests took 30 seconds to 1 min where as disconnected tests took 5 seconds or less. There is nothing else running on the host computer which could have an impact.

It's good that the results are the same but still wondering why it takes longer when connected if it should have all the data it needs?

Thanks
Tim

[NinjaTrader_ChelseaB]

Hi Tim,

I apologize for that. I had not considered that you may not be using the american date format.

Yes, June 10th 2013 would be a proper end date.

The data may be re-downloaded because of the rules for re-downloading historical data.

1. The End date parameter of the Data Series window contains the current day
2. The End date parameter of the Data Series window contains the last day of data available in your data repository
3. The data repository contains no data within 3 days of the first day loaded in the chart

Below is a link to the help guide on Historical & Real-Time Data. Please read the section 'When does NinjaTrader load historical data?'.
http://www.ninjatrader.com/support/h...rical_data.htm


If this becomes an issue, you can disable this so that data is not downloaded when you do your backtest. To do this:

• Click Tools -> Options... -> select the Data tab -> uncheck "Get data from server (if available)"

Please let me know if this does not resolve your inquiry.

[tdouglas]

That sounds fine, thanks for your help.

Tim