Strategies giving different results on VPS

[LightWeight]

Hello,

I recently set up an instance of Ninja on a VPS to do backtesting and optimization runs, and for some resaon I can't determine it is kicking out wildly different results than all my other machines. I have deployed Ninja and these algos on several local machines, as well as several other VPS instances, and the results are always very in line/matching. This is a new company/server (Vultr) and nothing I've checked has fixed it -- all of the indicators, algos, and type files have been copied and pasted exactly the same as from other machines, I've uninstalled and reinstalled Ninja, I've tried playing with different time zone settings, reloading historical data -- nothing works. When I connect to the live data feed the chart and price data all display correctly, but the minute I do an optimization or backtest the results are just dramatically different from every other machine/VPS, even though it should be using exactly the same data and logic. I'm now at a complete loss as to why the results are so different and wondering if there is anything else I should check that could be affecting the logic/results.

Regards,

Andrew

[NinjaTrader_ChelseaB]

Hello Andrew,

Start with a simple backtest of the SampleMACrossOver on both computers over the same time frame. (Same from and too date with the same bar type)

Have both computers set to the same time zone for simplicity.

Do both computers have the same results?
(If not, the data is different or one of the settings is different)

Are you obtaining your data from the same connection on both computers?

[LightWeight]

Hello Andrew,

Start with a simple backtest of the SampleMACrossOver on both computers over the same time frame. (Same from and too date with the same bar type)

Have both computers set to the same time zone for simplicity.

Do both computers have the same results?
(If not, the data is different or one of the settings is different)

Are you obtaining your data from the same connection on both computers?

I tried that, and the results came back differently again. Both computers are using the same settings, same time zone, and same data feed. I've attached a screenshot showing the results for both servers for comparison, for a 1 minute test. I then ran the test again with a 30 minute bar setting, and again the results were different. What I noticed in looking at the actual trades though is that while the recent trades match perfectly, in terms of entry time and price, the earliest trades are different -- I've attached a second screenshot showing this.

So again, I'm not sure what the problem is, everythign should be exactly the same, and yet as you can see the results over time in the past are slightly different, which means that the net result for all trades is quite different. I don't see how that's possible when both servers are using the same data feed and settings.

[NinjaTrader_ChelseaB]

Hello LightWeight,

Thank you for confirming that both computers are connecting to the same data feed.

Though they are both using the same data feed, if the results of the sma backtest is different then the data is different.

What is the Merge policy set to on both machines?

I would like you to use the StreamWriter reference sample on both computers so that I may compare the historical data.
http://www.ninjatrader.com/support/f...ead.php?t=3475

This reference sample will write a file with all of the historical bar values (OHLC) so that we can compare the two machines and see what is different.
The will be named MyTestFile.txt and will be located in the Documents\NinjaTrader 7\ folder.

Please send the MyTestFile.txt from both computers tested over the same time frame, instrument, and bar type as the strategy on both machines.

[LightWeight]

Hello LightWeight,

Thank you for confirming that both computers are connecting to the same data feed.

Though they are both using the same data feed, if the results of the sma backtest is different then the data is different.

What is the Merge policy set to on both machines?

I would like you to use the StreamWriter reference sample on both computers so that I may compare the historical data.
http://www.ninjatrader.com/support/f...ead.php?t=3475

This reference sample will write a file with all of the historical bar values (OHLC) so that we can compare the two machines and see what is different.
The will be named MyTestFile.txt and will be located in the Documents\NinjaTrader 7\ folder.

Please send the MyTestFile.txt from both computers tested over the same time frame, instrument, and bar type as the strategy on both machines.

I downloaded and installed on both servers and ran a backtest again, but I don't see this file anywhere in the Ninja folders. A search didn't turn it up either.

[NinjaTrader_ZacharyG]

Hello LightWeight,

Have you imported the StreamWriter reference sample indicator ZIP file?

Once you have imported it, add this indicator to the backtest results chart.

This reference sample will write a file with all of the historical bar values (OHLC) so that we can compare the two machines and see what is different.

The will be named MyTestFile.txt and will be located in the Documents\NinjaTrader 7\ folder.

Please send the MyTestFile.txt from both computers tested over the same time frame, instrument, and bar type as the strategy on both machines.

[LightWeight]

Hello LightWeight,

Have you imported the StreamWriter reference sample indicator ZIP file?

Once you have imported it, add this indicator to the backtest results chart.

This reference sample will write a file with all of the historical bar values (OHLC) so that we can compare the two machines and see what is different.

The will be named MyTestFile.txt and will be located in the Documents\NinjaTrader 7\ folder.

Please send the MyTestFile.txt from both computers tested over the same time frame, instrument, and bar type as the strategy on both machines.

Ok I have them now, files are attached.

Edit: don't know if you needed the settings, but this was a SampleMACrossover backtest on CL 01-16, from 1/1/2009 to 12/15/2015, using a 15 min bar, no slip or comish.

[NinjaTrader_ZacharyG]

Hello LightWeight,

What is the Merge policy set to on both machines?

You can find this setting by clicking on Tools -> Options -> Data tab.

The Merge policy will be under the "Historical chart data" section.

[LightWeight]

Hello LightWeight,

What is the Merge policy set to on both machines?

You can find this setting by clicking on Tools -> Options -> Data tab.

The Merge policy will be under the "Historical chart data" section.

They are both MergeBackAdjusted.

[NinjaTrader_ZacharyG]

Hello LightWeight,

It looks like there are differences in data between the two files you have provided me. The "bad" server is missing data from "20151215 36.29 36.57 36.2 36.45" to "20151215 36.25 36.41 36.16 36.28".

Can you please shut down NinjaTrader and navigate to (My) Documents\NinjaTrader 7\db directory on the "bad" server? Navigate into the day, minute, and tick directories and delete folders pertaining to CL.

Once you have done this, restart NinjaTrader, connect to the same data feed as before and attempt a backtest once more.

[LightWeight]

Hello LightWeight,

It looks like there are differences in data between the two files you have provided me. The "bad" server is missing data from "20151215 36.29 36.57 36.2 36.45" to "20151215 36.25 36.41 36.16 36.28".

Can you please shut down NinjaTrader and navigate to (My) Documents\NinjaTrader 7\db directory on the "bad" server? Navigate into the day, minute, and tick directories and delete folders pertaining to CL.

Once you have done this, restart NinjaTrader, connect to the same data feed as before and attempt a backtest once more.

I did that, and the backtest results are still coming back different like before.

[NinjaTrader_ZacharyG]

Hello LightWeight,

Can you verify that you are using the same session template on both servers? If both are using <Use instrument settings>, verify the CL 01-16 instrument on both servers, under Tools -> Instrument Manager are set to the same session template and that both servers have the same session template settings under Tools -> Session Manager.

Additionally, please navigate to the (My) Documents\NinjaTrader 7\db\minute directory on both servers, create a ZIP file of all folders in this directory pertaining to CL and send these ZIP files so we can investigate the historical data.

[LightWeight]

Hello LightWeight,

Can you verify that you are using the same session template on both servers? If both are using <Use instrument settings>, verify the CL 01-16 instrument on both servers, under Tools -> Instrument Manager are set to the same session template and that both servers have the same session template settings under Tools -> Session Manager.

Additionally, please navigate to the (My) Documents\NinjaTrader 7\db\minute directory on both servers, create a ZIP file of all folders in this directory pertaining to CL and send these ZIP files so we can investigate the historical data.

Okay it looks like that was the issue -- I forgot that the session templates for indices and energy had changed recently with the earlier close time, and had not manually edited them to reflect that (and match the "good" server). Once I edited and saved the proper session end times and ran the tests again, they are now matching exactly as I was expecting them to, so I think the issue is solved. Thanks so much for your help.

[LightWeight]

Hello LightWeight,

Can you verify that you are using the same session template on both servers? If both are using <Use instrument settings>, verify the CL 01-16 instrument on both servers, under Tools -> Instrument Manager are set to the same session template and that both servers have the same session template settings under Tools -> Session Manager.

Additionally, please navigate to the (My) Documents\NinjaTrader 7\db\minute directory on both servers, create a ZIP file of all folders in this directory pertaining to CL and send these ZIP files so we can investigate the historical data.

Okay I may have spoken too soon -- our strategies for indices, energy and metals appear to be matching now, but for crops and currencies they are still giving completely different results. However I checked the session templates for both groups and they appear correct, so again I'm confused why the results are coming back different.

[NinjaTrader_ZacharyG]

Hello LightWeight,

Can you utilize the same method with the sample indicator on both the "good" and "bad" servers so we can compare the historical data?

Hello LightWeight,

Have you imported the StreamWriter reference sample indicator ZIP file?

Once you have imported it, add this indicator to the backtest results chart.

This reference sample will write a file with all of the historical bar values (OHLC) so that we can compare the two machines and see what is different.

The will be named MyTestFile.txt and will be located in the Documents\NinjaTrader 7\ folder.

Please send the MyTestFile.txt from both computers tested over the same time frame, instrument, and bar type as the strategy on both machines.