Interactive Brokers slow data load

[zweistein]

All NT8 beta versions have this problem:

Data loading is extremly slow:
Example 2 charts, FDAX 09-16 1 min and FDAX 09-16 weekly, each chart 1000 bars.

When connecting to IB data loading takes several minutes (something between 5 and 10 minutes) .
In comparison NT7 takes less than 3 minutes.
Data loading from within IB TWS is immediate (less than 5 seconds)


Conclusion: NT8 beta 12 is not useable for trading yet.

Looks like it is time to look for software alternatives.

@NT : Are you silently dropping support for IB?

Best regards

Andreas

[NinjaTrader_ChristopherJ]

Hello,

Thank you for your post.

I am not aware of Interactive Brokers being dropped as a supported provider/brokerage.

So I may look into this further please go to Help->Email Support and be sure to include Attn: Chris Jameson in the Subject line, a link to this thread in the body and a check in the box for "Log and trace files".

[marcel-nl]

Hello,

Thank you for your post.

I am not aware of Interactive Brokers being dropped as a supported provider/brokerage.

So I may look into this further please go to Help->Email Support and be sure to include Attn: Chris Jameson in the Subject line, a link to this thread in the body and a check in the box for "Log and trace files".

The earlier response from NT was that this issue would not be solved before release of NT8. It would be fixed in an early version after go Live. I would also like to remember that there is also an outstanding and confirmed issue with data spikes with the IB data feed which did not occur in NT7. So I am also concerned with the level of support regarding IB.

[NinjaTrader_ChristopherJ]

Do you happen to have a link to the thread that covered this or a ticket number with support you can provide?

[marcel-nl]

http://ninjatrader.com/support/forum/showthread.php?t=85252&highlight=marcel-nl

http://ninjatrader.com/support/forum/showthread.php?t=81792&highlight=marcel-nl

[NinjaTrader_ChristopherJ]

As a test can you please start NinjaTrader in Safe Mode.
Safe Mode will prevent NinjaTrader from:

• Loading workspaces.
• Connecting on start-up.
• Loading custom assemblies.
• Getting instrument updates from the server.

To enable safe mode, please use the following steps:

• Exit NinjaTrader.
• Hold the CONTROL key and double click the NinjaTrader icon.
• Keep the CONTROL key held down until you see the Control Center.
• You can verify you are in safe mode by going to Help -> About.

Next connect to IB and let me know if you see a faster load time.

[marcel-nl]

As a test can you please start NinjaTrader in Safe Mode.
Safe Mode will prevent NinjaTrader from:

• Loading workspaces.
• Connecting on start-up.
• Loading custom assemblies.
• Getting instrument updates from the server.

To enable safe mode, please use the following steps:

• Exit NinjaTrader.
• Hold the CONTROL key and double click the NinjaTrader icon.
• Keep the CONTROL key held down until you see the Control Center.
• You can verify you are in safe mode by going to Help -> About.

Next connect to IB and let me know if you see a faster load time.

In the past the issue has been investigated. NT connected to my platform and have done own testing. The reaction was that there need to be a redesign of the IB connector and there was currently no capacity for it. I have stopped testing NT8 waiting for the solution (and solving the issue with data spikes , which is also a show stopper for me).

[DaveE]

Hello Andreas,

I think that in NT8 historical data request always seems to get both Bid and Ask from IB.
Whereas in NT7 it only got one or the other.

I think this means twice as many IB API historicalData() callbacks.
If these IB callbacks are slow this might double the time to load the same period.
Also there's probably twice as many calls to write to the NT database.

[zweistein]

very slow IB data load

OK, so I have 2 charts open. Each chart has 700 Data bars.

Instrument is FDAX 09-16.

One is a 1 Minute chart, the other is a weekly chart.

The 1 minute chart loads in less than 5 minutes, still much slower than IB TWS charting which is almost immediate.

The weekly chart is still loading, and more than 50 minutes have passed now... Unuseable.

It seems obvious that whenever there is a bar loading problem ( bar not in data base, etc) then NT8 unwinds all the exceptions generated. But this unwinding the exceptions is much too slow, it cannot take more than 45 minutes for 700 bars!

Also, once the data is loaded there is no need to reload everything on a reconnect.


P.S. Have send you workspaces etc. in the past already on the same problem in early NT8 beta versions.


Andreas

[NinjaTrader_PatrickG]

OK, so I have 2 charts open. Each chart has 700 Data bars.

Instrument is FDAX 09-16.

One is a 1 Minute chart, the other is a weekly chart.

The 1 minute chart loads in less than 5 minutes, still much slower than IB TWS charting which is almost immediate.

The weekly chart is still loading, and more than 50 minutes have passed now... Unuseable.

It seems obvious that whenever there is a bar loading problem ( bar not in data base, etc) then NT8 unwinds all the exceptions generated. But this unwinding the exceptions is much too slow, it cannot take more than 45 minutes for 700 bars!

Also, once the data is loaded there is no need to reload everything on a reconnect.


P.S. Have send you workspaces etc. in the past already on the same problem in early NT8 beta versions.


Andreas

700 bars of weekly data would be more than 10 years of historical data which may run into problems with Interactive Broker's pacing violation.

I would like to look into your diagnostic files to see if this is what is occurring,

Please send me your log and trace files so that I may look into what occurred.

You can do this by going to the Control Center-> Help-> Mail to Platform Support.

Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

Please reference the following ticket number in the body of the email: 1543111 ATTN Patrick G

[zweistein]

Please prepare me a simple workspace with FDAX 09-16 with 2 charts, 1 min and weekly with each 700 bars.
Then test and if you cannot find slow data load then please post the workspace here so that me and others can cross check your findings.

As soon as I have a workspace from you I will report my findings.
I want a workspace from you so that there can be no misunderstanding .



Best regards

Andreas

[NinjaTrader_PatrickG]

Please prepare me a simple workspace with FDAX 09-16 with 2 charts, 1 min and weekly with each 700 bars.
Then test and if you cannot find slow data load then please post the workspace here so that me and others can cross check your findings.

As soon as I have a workspace from you I will report my findings.
I want a workspace from you so that there can be no misunderstanding .

Best regards

Andreas

I believe the long load time might be expected due to the current behavior of the IB API and NinjaTrader. IB only allows 60 historical data chunks to be requested every 10 minutes. A chunk of historical data roughly equates to 2 days of minute data. The IB adapter attempts to queue chunk requests at a rate of 5 seconds per request. This is in an attempt to prevent hitting the pacing violation. With this information, it is easy to see why large historical requests can take time when using IB.

I experience long load times as well. The minute bar loads relatively quickly. The 700 bar weekly chart is over 10 years of data and did not load in the time I was able to test. Providing a workspace would not get us anywhere at this point.

I would like to see, in detail, what Interactive Brokers is reporting in your NinjaTrader trace file so I can match it up with the information from my testing.

Please send me your log and trace files so that I may look into what occurred.

You can do this by going to the Control Center-> Help-> Mail to Platform Support.

Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

Please reference the following ticket number in the body of the email: 1543111 ATTN Patrick G

[zweistein]

well,

send me the workspace and I will then test it.

when I disconnect and reconnect to IB , then there is at maximum 3 minutes of data lost.
Loading time is then also extremly slow.

So, please post a workspace here that other users can then give you feedback also

Andreas

[Pete Collins]

You need to switch to Kinetick. I use IB as my broker but the data feed into Ninjatrader is just so slow and unusable. I think its their way of trying to get people to use TWS instead of Ninjatrader. Just go with Kinetick for charting data and then use TWS to place trades...

[NinjaTrader_PatrickG]

well,

send me the workspace and I will then test it.

when I disconnect and reconnect to IB , then there is at maximum 3 minutes of data lost.
Loading time is then also extremly slow.

So, please post a workspace here that other users can then give you feedback also

Andreas

My workspace is not performing any better than your workspace when I recreate it as you described in your earlier post. Me providing a workspace would not be a useful tool for testing.

As Pete Collins mentioned, the IB API has certain restrictions that other providers do not.

Please consider the information I've shared regarding the IB API and send me the requested information.