Interactive Brokers slow data load

[marcel-nl]

As Pete Collins mentioned, the IB API has certain restrictions that other providers do not.

It is my understanding that the problem is not the IB API per se, but the way Ninja Trader handles it. Earlier it was stated by Ninja Trader that they currently do not have the capacity to change the Ninja Trader connector in order to have a better performance. " There are no changes to the data load rules in Beta 8. If you are referring to the amount of time it takes to return large data request, we have no immediate plans during beta to change these data load rules due to the pacing policies used by the provider. We may revisit this in the future as it's highly demanded, but will take some conceptual changes that we just cannot accommodate right now. "

I would like to have a very clear statement of Ninja Trader whether they are committed to supporting the IB data feed and will solve the issue with the slow data load. And I would like to know whether/when they will solve the issue with the price spikes (see link below).
http://ninjatrader.com/support/forum...ight=marcel-nl

[NinjaTrader_PatrickG]

It is my understanding that the problem is not the IB API per se, but the way Ninja Trader handles it. Earlier it was stated by Ninja Trader that they currently do not have the capacity to change the Ninja Trader connector in order to have a better performance. " There are no changes to the data load rules in Beta 8. If you are referring to the amount of time it takes to return large data request, we have no immediate plans during beta to change these data load rules due to the pacing policies used by the provider. We may revisit this in the future as it's highly demanded, but will take some conceptual changes that we just cannot accommodate right now. "

I would like to have a very clear statement of Ninja Trader whether they are committed to supporting the IB data feed and will solve the issue with the slow data load. And I would like to know whether/when they will solve the issue with the price spikes (see link below).
http://ninjatrader.com/support/forum...ight=marcel-nl

We received concerns from many users in NinjaTrader 7 about experiencing pacing violations. Steps were put in place to prevent pacing violations, however, this now results in longer data load times. I explained this in further detail in a previous post in this thread. This is absolutely a limitation of how Interactive Brokers chooses to supply their data through the API. If users would rather get pacing violations, a change may be taken into consideration. I can add your vote for this change. There is no ETA for if/when a change would be implemented.

As for the data spikes, this is caused by "unreportable trades" being sent from Interactive Brokers to NinjaTrader. Product Management has spoken with Interactive Brokers in depth regarding this data. Currently, these trades cannot be identified when being sent to NinjaTrader and cannot be filtered out. Client's will need to set up a bad tick filter to help prevent these. no change will occur on the NinjaTrader side as IB sends no way to identify these orders.

[marcel-nl]

We received concerns from many users in NinjaTrader 7 about experiencing pacing violations. Steps were put in place to prevent pacing violations, however, this now results in longer data load times. I explained this in further detail in a previous post in this thread. This is absolutely a limitation of how Interactive Brokers chooses to supply their data through the API. If users would rather get pacing violations, a change may be taken into consideration. I can add your vote for this change. There is no ETA for if/when a change would be implemented.

As for the data spikes, this is caused by "unreportable trades" being sent from Interactive Brokers to NinjaTrader. Product Management has spoken with Interactive Brokers in depth regarding this data. Currently, these trades cannot be identified when being sent to NinjaTrader and cannot be filtered out. Client's will need to set up a bad tick filter to help prevent these. no change will occur on the NinjaTrader side as IB sends no way to identify these orders.

Regarding the first topic (slow data load). The earlier response from Ninja Trader product management was that conceptual changes are needed to the design of the Ninja Trader connector and there currently is no capacity for making this change (see the quote provided before). Now you are saying that a solution is not possible and that a choice has to be made between pacing violation or slow data load times. This conflicting statements by Ninja Trader make me only more concerned.

Regarding the second topic. The problem does not occur in NT7 (as also confirmed by Ninja Trader). http://ninjatrader.com/support/forum...ight=marcel-nl

"I have been testing throughout the day and I have replicated this behavior when viewing SPY in NinjaTrader 8 using TWS 944.3 and 954, but could not replicate in NinjaTrader 7.

I will submit this to the Development Team so they can begin working on a resolution. "


The bad tick filter option of NT8 does not solve the problem. So this is a show stopper

[zweistein]

Ok, as you can replicate the problem please go forward to fixing it.

You do not need to know rocket science for this:

Solution 1:

For example, just add an option like :
Historical Request lookback: 1 week
1 month
maximum


Then you would have resolved the problem.

The user then downloads historical data only one time. Older data would be loaded from the locally stored db of NT.

For all future connections the user would select 1 week lookback and everything would work smoothly.


Solution 2:

A variant of the above:

Initially load historical data only from the local database (all data in the NinjaTrader 8/db directory .
Download only historical data from the last connection until now.


Solution 3:

Variants of above, but then start a background thread that respects the IB data download imitiations.

Please be reminded that under normal useage, this means open NinjaTrader 8 at least once a week and connect to IB you will never run into any IB data limitations as the data requested will be very little and below any IB quantity limitations.


Unfortunately the NinjaTrader 8 software is so dumb that it wants to downlaod ALL historical data each time , even if it is 5 or 10 years old.


Did you get my point?


Best regards

Andreas

[NinjaTrader_PatrickG]

Thanks for your feedback. It will be taken into consideration.

[marcel-nl]

As for the data spikes, this is caused by "unreportable trades" being sent from Interactive Brokers to NinjaTrader. Product Management has spoken with Interactive Brokers in depth regarding this data. Currently, these trades cannot be identified when being sent to NinjaTrader and cannot be filtered out. Client's will need to set up a bad tick filter to help prevent these. no change will occur on the NinjaTrader side as IB sends no way to identify these orders.

It was not a problem in NT7 as confirmed by Ninja Trader. When I did the testing when raising the issue the bad tick filter did not prevented the data spikes. So for me it is a Show Stopper to use NT8 and I stopped further testing. Please find the original thread. I am not convinced that it will be workable without improvements to NT8.

http://ninjatrader.com/support/forum/showthread.php?t=85252&highlight=marcel-nl

[NinjaTrader_PatrickG]

It was not a problem in NT7 as confirmed by Ninja Trader. When I did the testing when raising the issue the bad tick filter did not prevented the data spikes. So for me it is a Show Stopper to use NT8 and I stopped further testing. Please find the original thread. I am not convinced that it will be workable without improvements to NT8.

http://ninjatrader.com/support/forum...ight=marcel-nl

I mentioned the cause and all the information I currently have on this behavior in my earlier replies in this thread.

Product Management is aware of this behavior, is tracking feedback on the new logic which was put in place to prevent pacing violations, and will take your feedback into consideration.

[NinjaTrader_Brett]

Hello,

For those of you using NinjaTrader 8 with IB and running into slower historical data loading and data spikes can you please contact me at platformsupport at ninjatrader dot com ATTN: Brett and reference this forum thread if you would be willing to test an advanced 8.0.2.0 build which holds some changes which should positively improve the situation.

-Brett