IB and pacing errors

[Steve R]

I know this has been gone over at length with many people, but I just want to clarify something here.

First off I have already downloaded and saved all the historical data for my charts.

I probably have 30? charts open and only one workspace - as I had previously seen that NT downloads data for instruments in workspaces that aren't even open. Having less charts/instruments open or opening 10 now, 10 later and maybe 10 more, 30 mins from now is not an option I'm going to entertain. This is 2012 after-all.

Now to what I think might be the solution but I need to confirm it with NT support in case my thinking is wrong.

I typically have no more than 4-5 days TOPS between opening and D/L "new" instrument data from IB - my only provider.

My "lookback" period has always been 200-250 days (I use 95% daily charts and maybe 1-3 intra-day) I realize this is the issue (200-250 day lookback).

The solution? - OK If I can be sure to not need more than 5-6 days of "new" data, should I set the charts to look back ONLY that far?

If so, then since I want to see 200 days back, would I be able to then reset the chart once I was connected and the "new" data D/L, to a 200-250 day period WITHOUT NT going back to IB and THEN requesting the 200-250 days all over again?

The issue here is I want to see 200-250 days but I clearly do not want to D/L ALL the data again. It would certainly be acceptable to D/L 5-6 days then reset to 200 if it ensured I did not get a pacing error since no new data would actually be required at that point.

Is my thinking correct here?

[NinjaTrader_Jason]

Hello Steve R,

Please see the link below under 'When does NinjaTrader load historical data?' below that explains when NinjaTrader loads historical data from the connected data feed.

http://www.ninjatrader.com/support/helpGuides/nt7/historical_data.htm

Yes, I suggest to keep the same value for the lookback period. This way you avoid requesting historical data before the first date you have historical data available in repository/cache. If you request historical data before this date, it will load all historical data anew. See #4 below.


Examples of when NinjaTrader will fetch data if the data repository contains data from 1/3/08 to 1/6/08 and the current date is 1/7/08:


1.Chart of 1/3/08 to 1/7/08 -> load data request for 1/7/08, use data stored in data repository/cache for other dates

2.Chart of 1/3/08 to 1/6/08 -> load data request for 1/6/08, use data stored in data repository/cache for other dates

3.Chart of 1/3/08 to 1/5/08 -> use data stored in data repository/cache for all dates

4.Chart of 12/28/07 to 1/5/08 -> load data request for all dates

[Steve R]

Hi Jason.

The reason I asked a specific question is because I am still not sure I understand after reading the material and watching NT "operate".

So the question is, assuming I have all historical data and only need the current (let's assume last 2 days) worth of data when I re-connect. If my days to load is set at 200, will it only go back to IB and request the last 2 days or will attempt to load all 200 again despite having the 'old' data cached?

I ask because it 'seems' like it tries to reload it ALL every time I connect to IB, giving me a pacing error. And yes, I have it set to save historical data. Again, it also 'seems' like if I set days to load to say 10 days, have it connect and load the chart, then switch to 200 days it seems as if only then it will not reload data we don't need to since we have it already.

The way NT seems to be operating looks different than what the material says it does and the pacing errors looks to confirm it.

"1.Chart of 1/3/08 to 1/7/08 -> load data request for 1/7/08, use data stored in data repository/cache for other dates"

This is how I originally thought it worked too. I'm going to have to figure out why it doesn't look like this is how it is operating.


4.Chart of 12/28/07 to 1/5/08 -> load data request for all dates

This may be a clue. I'll have to go back and check. IB does not store a lot of historical data. If I continually ask for data far enough back and it never 'gets' it because it isn't there, my guess from reading this that it WOULD attempt to reload ALL data every time I ask regardless, since I'm always asking for data that I really don't have in the cache but it's far enough back that I haven't actually noticed.

I think my problem is my 'assumption' that I have all the data when it's possible that I don't due to IB's lack of historical cache. For a long time I will be asking for more historical than they have, causing a complete reload every time. When MY cache finally obtains enough data as time advances on the calendar I suspect the problem would cease.

[NinjaTrader_Jason]

Hello Steve R,

It will only load the last two days. As long as you do not request data before the first date available in the repository/cache, it will not load all historical data from the connected data feed.

Could it be the pacing error is caused by something else? Do you perhaps have workspaces opened in the background that could be the cause? Maybe you have custom/3rd party indicators/strategies applied that request additional historical data and cause the pacing error?

If the issue remains, please proceed as follows. Shut down NinjaTrader and navigate to the following folder:

C:\-->Users-->*Windows user account*-->Documents-->NinjaTrader 7-->db

Right click the cache folder and select 'Cut'. Copy the folder temporarily to your desktop.

Navigate to the following folder:

C:\-->Users-->*Windows user account*-->(My) Documents-->NinjaTrader 7-->workspaces

Delete the _Workspaces.xml file. This way Ninjatrader does not open any workspaces once started - it will open an empty Untitled1 workspace instead. This will not delete your workspaces.

Start NinjaTrader and connect to IB.

Create a new chart. Check if all data is loaded and save the chart in a workspace.

Restart NinjaTrader and check what happens when you connect to IB.

[Steve R]

Hi Jason,

"As long as you do not request data before the first date available in the repository/cache, it will not load all historical data from the connected data feed."

That was it. I just scaled back the 'days to load' to ensure I was not repeatedly requesting data that IB could never deliver, and we nailed it perfectly with no pacing errors.

Once I cache a lot more data I would be able to get back to 220 days or so and it won't be a problem. Since a lot of what I view is futures data there are times where the chart will only have 60 days worth and me asking repeatedly for 200 caused a reload of the entire amount, every day. Since I also do spread trading using multi-instrument charts there are also times where one contract has only 80 days yet the other has 180 days, so once again I need to load accordingly.

Thanks Jason.