IB - Historical data request pacing violation

[Style]

I'm getting the following error when attempting to get historical data from Interactive Brokers:

"Ninja Trader has reached an IB limitation for historical data requests. IB may reject all historical data requests for the next 5 minutes: historical Market Data Service error message: Historical data request pacing violation"

I contacted IB support, and they said this happens when you make more than 60 requests in 10 minutes. Note that I'm getting this error when just bringing up ONE chart! I bring up one chart for an ETF, bar size 5 minutes, and the farthest back I can go is 5 months. If I select farther than 5 months, then I get the error above. IB tells me that I should have access to 12 months of data:

http://www.interactivebrokers.com/en/software/api/apiguide/api/historical_data_limitations.htm

So it seems to me that even though I'm loading just one chart, NT is making multiple requests, such that it makes about 60 requests to load 5 months of intra-day data. Is there a way to get around this?

[NinjaTrader_Kyle]

Hi,

Thanks for your post.

This is not behavior which we would expect to see when only a single chart is used.

Please contact me via email at Support AT NinjaTrader DOT com.

In the message, please include a link to this forum thread and a copy of today's trace file. The file I need would be named 'trace.20130403.txt'.

[Style]

IB tells me that I should have access to 12 months of data:

http://www.interactivebrokers.com/en...imitations.htm

Just reading through that link closely, and there are more than one conditions listed for the pacing violation:

The following conditions can cause a pacing violation:

• Making identical historical data requests within 15 seconds;
• Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds.

Also, observe the following limitation when requesting historical data:

• Do not make more than 60 historical data requests in any ten-minute period.

Could this also be related?

Each request is restricted to duration and bar size values that return no more than 2000 bars (2000 bars per request).

5 months is roughly 105 trading days. During regular trading hours there are 78 bars for a 5 minute period, which equals about 8190 bars. If restricted to just 2000 bars per request, this would still only be 5 requests.

However, even though my chart is set to 5 min bars it appears that NT collects 1 minute data. I can tell this by connecting to IB, getting data via a 5 minute chart, disconnecting, and then bringing up a 1 minute chart. Still, this means that for five months instead of 8190 5 minute bars it's getting 40950 1 minute bars. Divide this by 2000 bars per request, and it should still be able to get 5 months of data with just 21 requests.

What gives? Is NT requesting something less than 2000 bars per request? If so, is there a way to change this?

Thanks!

[Style]

Email sent. Thanks Kyle!

[astrolobe]

Did this ever get resolved? I am having the same problems.

[NinjaTrader_Kyle]

Hello astrolobe,

Please contact me via email at Support AT NinjaTrader DOT com.

In the message, please include a link to this forum thread and a copy of today's trace file. The file I need would be named 'trace.20130929.txt'.

[mjeebusbox]

IB - Historical data request pacing violation

I'm having the exact same problem with NT 7 connected to IB.

"Ninja Trader has reached an IB limitation for historical data requests. IB may reject all historical data requests for the next 5 minutes: historical Market Data Service error message: Historical data request pacing violation"

I have 5 charts on my screen:

- Weekly: Pulling 365 days
- Daily: Pulling 365 days
- 240 min: Pulling 365 days
- 30 min: Pulling 365 days
- Leaders and Laggers chart: With 4 market ETF's (SPY, IWM, DIA, IWM, QQQ), pulling 5 days of data.

Can someone please help me also?

[NinjaTrader_PatrickH]

Hello mjeebusbox,

Thank you for your post.

Unfortunately, there is no workaround for the limitation to request data from Interactive Brokers. You will need to reduce the amount of data being requested at one time.

[raintrader]

Working around pacing errors.

I'm hoping that by understanding how Ninja Trader requests and stores historical data I will be able to adjust my usage to prevent the errors from occurring. I've tried the following:

I opened one chart and allowed the chart to fully populate with historical data. Once that was done I waited 15 minutes and then opened a second chart. At this point, if I understand the data feed mechanism correctly, I should be making no additional historical data requests for that first chart and only making requests for the newly opened 2nd chart.

IB says not to make more than 60 historical data requests in a 10 minute window. I was able to load about 1.5 years of data (240 min bars) for the first chart, but I am unable to load even 50 weeks of data without an error when I open a 2nd chart.

In addition, despite showing on the lower right bottom of the control window that most of the requested data has been downloaded, all that data seems to be lost and only one bar is shown on the chart after the error message pops up.

Three questions:

1. Are additional historical data requests being made after all historical data has already been downloaded for a open chart?

2. Is there a way to actually display and use the partial data that has been downloaded when a pacing error does occur rather than lose it all and have to try again. (I often get 80% of the data I've requested before an error occurs.)

3. If a single chart is opened and historical data is downloaded one chart at a time, will that data be saved to the disk and future data requests be reduced to only what is needed when the chart is reopened?

Thanks

[NinjaTrader_PatrickH]

Hello raintrader,

Thank you for your post and welcome to the NinjaTrader Support Forum!

1. Are additional historical data requests being made after all historical data has already been downloaded for a open chart?

NinjaTrader loads data from your data provider whenever it determines it could potentially not have all the data pertaining to the requested time period. Charts that have already downloaded historical data for the period you have requested should not need to request that data again.

2. Is there a way to actually display and use the partial data that has been downloaded when a pacing error does occur rather than lose it all and have to try again. (I often get 80% of the data I've requested before an error occurs.)

3. If a single chart is opened and historical data is downloaded one chart at a time, will that data be saved to the disk and future data requests be reduced to only what is needed when the chart is reopened?

Any data that has been downloaded should be stored in the Historical Data Manager. If you disconnect from your provider and open a new chart of the instrument it should load any available data.

In addition, you could disable the load of data from the server by going to Tools > Options > Data > disable the option for 'Get data from server' > OK. This will disable downloading historical data from the provider on the charts. However, you can then go to Tools > Historical Data Manager > Download, and download the needed data (of source within the confines of the restricted amount of data you can download).

Please take a look at the following links for more information:

• When NinjaTrader downloads historical data: http://www.ninjatrader.com/support/h...rical_data.htm
• Information on the Historical Data Manager: http://www.ninjatrader.com/support/h...ta_manager.htm

Please let me know if you have any questions.

[raintrader]

That information has helped me reduce the severity of the problem. Now if I can just figure out why IB sometimes allows a historical data request for a certain amount of data, and other times will not. But that's a question for a different company and a different forum.

Thanks Patrick.

[schematics]

You may want to take a look at this downloader. It saved me countless hours of coding, working through various IB API historical data limitations and other quirks. The UI is simple, and works for most contracts you can trade through Interactive Brokers, and best of all -takes care of pacing violations automatically.

http://www.tradinggeeks.net/downloads/ib-data-downloader/