Why is backtesting redownloading all my data?

[cunparis]

I notice when I disconnect from my data provider, backtesting is very fast. Seconds. When I'm connected to the data (IQFeed), it's incredibly slow. Like 30-60 seconds for a backtest.

Is Ninjatrader trying to download the data it already has?

[NinjaTrader_Ben]

Hello,

Thank you for your note.

If you backtest on more data than is already stored in your data base NT will pull from the servers which will take more time. If you work off of the data that is already in your data base it will go faster. If you disconnect you can only work off or you database so it is always fast.

This link will help:

http://www.ninjatrader-support.com/HelpGuideV6/WhenDoesNinjaTraderLoadHistoricalData.html

[cunparis]

Hello,

Thank you for your note.

If you backtest on more data than is already stored in your data base NT will pull from the servers which will take more time. If you work off of the data that is already in your data base it will go faster. If you disconnect you can only work off or you database so it is always fast.

This link will help:
http://www.ninjatrader-support.com/H...ricalData.html

Thanks for the explanation. I think there was a problem because after a few more backtests it got stuck for 15+ minutes so I exited and restarted and then it worked in seconds as usual.

[cunparis]

Thanks for the explanation. I think there was a problem because after a few more backtests it got stuck for 15+ minutes so I exited and restarted and then it worked in seconds as usual.

It's happened again. Just freezes. I have to exit and then redo the backtest and it's really fast.

Any log I could look at to see if there are problems?

[NinjaTrader_JoshP]

Try the trace or log folder in My Documents\NinjaTrader 6.5.

[cunparis]

I have some new info:

I run a backtest for a one year period (2007). It takes about 10 minutes for it to load the data.

I modify my strategy a bit and re-run the test. It takes 10 minutes again to load the data.

The 2nd time I run, I should have all the data. I don't see why ninjatrader has to download the data again. IQ is unusally slow today which is making the problem really bad.

I know I can disconnect and run my backtests, but I don't see why I should have to do that. NT shouldn't even attempt to download data if I already have it right?

[NinjaTrader_Ben]

Hello,

I suggest openning your task bar and taking a look at your CPU usage (ctrl+alt+delete).

Also, how many instruments are what type of data are you using? What changes are you making to your strategy? Does this happen when you don't modify your strategy?

[cunparis]

Hello,

I suggest openning your task bar and taking a look at your CPU usage (ctrl+alt+delete).

Also, how many instruments are what type of data are you using? What changes are you making to your strategy? Does this happen when you don't modify your strategy?

Yes I don't modify my strategy, I just re-run it and I get the "Loading Data..." box for 5+ minutes. This is for 1 year data on a 5 minute chart of ES. No other instruments. if I disconnect then of course it's fast. Which is what I usually have to do.

[cunparis]

PS: I have a quad core and it seems Ninjatrader is only using one CPU. When the CPU is around 75% busy..

[cunparis]

this is happening to me non-stop. It's totally unusable.
I recently upgraded, I don't know if that has something to do with it.
I run the same strategy several times, with IQFeed connected it takes 5 minutes or more, without a dconnection it takes seconds. There's obviously something wrong, it's looking for data from IQ and timing out or something I don't know..

[cunparis]

Here is the trace: I disconnected at 17:01 because I couldn't take it any more. That makes NT crash, which is another problem.

2008-10-30 16:51:15:551 (IQ Delta) Cbi.ConnectionStatusEventArgs.Process: Connected Connected
2008-10-30 16:51:15:902 (IQ Delta) Data.Bars.GetBarsNow: instrument='QM 12-08' from='2008-02-21' to='2008-10-30' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2008-10-30 16:53:46:728 WARNING: Starting Strategy
2008-10-30 16:53:46:736 WARNING: Starting Strategy
2008-10-30 16:53:57:303 WARNING: Starting Strategy
2008-10-30 16:53:57:306 WARNING: Starting Strategy
2008-10-30 16:54:03:916 (IQ Delta) Data.Bars.GetBarsNow: instrument='ES ##-##' from='2006-12-29' to='2007-01-01' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2008-10-30 16:56:36:478 (IQ Delta) Data.Bars.GetBarsNow: instrument='ES ##-##' from='2006-12-29' to='2007-01-01' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2008-10-30 17:01:30:956 (IQ Delta) Cbi.Connection.Close1
2008-10-30 17:01:30:956 (IQ Delta) Cbi.Connection.Close2
2008-10-30 17:01:30:956 (IQ Delta) Cbi.Connection.Close3
2008-10-30 17:01:30:956 (IQ Delta) Cbi.Connection.Close4
2008-10-30 17:01:30:956 (IQ Delta) Cbi.Connection.Close5
2008-10-30 17:01:30:956 (IQ Delta) Cbi.Connection.Close6
2008-10-30 17:01:30:956 (IQ Delta) Dtn.Adapter.Cleanup
2008-10-30 17:01:30:966 (IQ Delta) Dtn.Adapter.DisconnectNow
2008-10-30 17:01:30:966 (IQ Delta) Cbi.Connection.Close9
2008-10-30 17:01:30:966 (IQ Delta) Cbi.Connection.ProcessEventArgs.ConnectionStatusEventArgs Disconnected Disconnected NoError 
2008-10-30 17:01:30:966 (IQ Delta) Cbi.ConnectionStatusEventArgs.Process: Disconnected Disconnected
2008-10-30 17:01:38:996 in OnUnhandledThreadException
2008-10-30 17:01:38:998 ********* exception trapped *********
2008-10-30 17:01:38:998 Object reference not set to an instance of an object.
2008-10-30 17:01:38:998    at NinjaTrader.Strategy.StrategyBase.Dispose()
   at NinjaTrader.Strategy.Strategy.Dispose()
   at NinjaTrader.Strategy.StrategyAnalyzer.Init()
   at NinjaTrader.Strategy.StrategyAnalyzer.OnRunBackTest(Object sender, EventArgs e)
   at System.Windows.Forms.MenuItem.OnClick(EventArgs e)
   at System.Windows.Forms.MenuItem.MenuItemData.Execute()
   at System.Windows.Forms.Command.Invoke()
   at System.Windows.Forms.Command.DispatchID(Int32 id)
   at System.Windows.Forms.Control.WmCommand(Message& m)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.TreeView.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
2008-10-30 17:01:48:165 Gui.WorkspaceOptions.ApplicationExit
2008-10-30 17:01:49:217 Gui.Globals.AppExit: shutting down ATI server
2008-10-30 17:01:49:218 Gui.Globals.AppExit: shutting down server
2008-10-30 17:01:49:219 Shutting down data server
2008-10-30 17:01:49:219 Gui.Globals.AppExit: shutting down SMTP server
2008-10-30 17:01:49:220 Gui.Globals.AppExit: closing connections
2008-10-30 17:01:49:220 Db.Globals.ApplicationExit
2008-10-30 17:01:49:231 Cbi.License.ApplicationExit
2008-10-30 17:01:49:231 Data.Collective2Adapter.ApplicationExit
2008-10-30 17:01:49:231 Checking backup reminder...
2008-10-30 17:01:49:234 ************************ Session End ************************

[NinjaTrader_JoshP]

Hi cunparis,

Do you have reproducible steps for how you got this trapped exception? If you could list them out exactly as you would do to reproduce it, it would help us evaluate the situation. Thank you.

As a side note, NinjaTrader does not currently use multi-cores.

[cunparis]

Hi cunparis,

Do you have reproducible steps for how you got this trapped exception? If you could list them out exactly as you would do to reproduce it, it would help us evaluate the situation. Thank you.

It seems to happen every time I disconnect from IQFeed while a backtest is in progress. It's happened several times. I have to exit NT and restart..

[cunparis]

This is the line from the trace, always the same:

2008-10-30 17:57:55:764 WARNING: Starting Strategy
2008-10-30 17:57:58:727 (IQ Delta) Data.Bars.GetBarsNow: instrument='ES ##-##' from='2006-12-29' to='2007-01-01' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0

Seems like a bug. The dates it's requesting are 12-29 to 01-01. I am testing my strategy from 1/1/07 to 1/1/08. I suppose it can't get the data for some reason but other than receive an error it just hangs..

[cunparis]

You can see here that I run the same strategy several times and it asks IQFeed for the TICK data each time, but not for ES. I didn't change the time period or anything else..

2008-10-30 18:10:39:364 WARNING: Starting Strategy
2008-10-30 18:11:22:639 (IQ Delta) Data.Bars.GetBarsNow: instrument='ES ##-##' from='2007-11-30' to='2007-12-01' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2008-10-30 18:12:13:900 (IQ Delta) Data.Bars.GetBarsNow: instrument='ES ##-##' from='2007-11-30' to='2007-12-01' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2008-10-30 18:13:50:118 (IQ Delta) Data.Bars.GetBarsNow: instrument='^TICK' from='2007-01-02' to='2007-12-01' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2008-10-30 18:14:56:425 WARNING: Calculated target price for strategy 'PivotTrader' was smaller/equal 0. No target order placed.
2008-10-30 18:16:05:305 WARNING: Starting Strategy
2008-10-30 18:16:05:309 WARNING: Starting Strategy
2008-10-30 18:16:07:577 (IQ Delta) Data.Bars.GetBarsNow: instrument='^TICK' from='2007-01-02' to='2007-12-01' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2008-10-30 18:16:38:375 WARNING: Calculated target price for strategy 'PivotTrader' was smaller/equal 0. No target order placed.
2008-10-30 18:17:49:575 WARNING: Starting Strategy
2008-10-30 18:17:49:580 WARNING: Starting Strategy
2008-10-30 18:17:51:116 (IQ Delta) Data.Bars.GetBarsNow: instrument='^TICK' from='2007-01-02' to='2007-12-01' sessionBegin='00:00:00' sessionEnd='00:00:00' excludeWeekend=False period=1 Min splitAdjusted=False dividendAdjusted=False bars=0