Interactive Brokers Historical Data Loads Slow

[mgjul]

I am just writing to share my thoughts about the recent NT8 release. First I want to say that I have been a NinjaTrader user for several years and never had anything to complain about really. I'm an active futures daytrader and I need a platform that is both efficient and fast. But after using NT8 for a week, I have to admit that I am very dissapointed by the new release. It is VERY slow to load, still buggy to some extent, doesn't deal well with live data from Interactive Brokers and there is that lack of compatibility between NT7 indicators, workspaces, etc that is a pain. So, for all those reasons I'm switching back to NT7 and suggest that NinjaTrader work harder on its new release to make it as good as NT7 otherwise it will lose a lot of customers. Just my 2 cents worth of advice.

[junkone]

i second it. but i also remember the days of NT7 rollout. it was buggy. hopefully the next few releases will make it better.

[NinjaTrader_Ray]

First of all, thank you for your many years of loyal support of the NinjaTrader platform. Feedback is an important part of how we can learn to improve our products. There is no doubt that there are still bugs however, I expect this to go down over time. With respect to the slow loading times, we would be interested to learn more as NinjaTrader 8 as a whole delivers much better performance than NinjaTrader 7. Script incompatibly is expected and unfortunately nothing will change there. This was a conscience decision made in order to push the boundaries of our development framework to allow the community to build richer and more integrated apps on top of the NinjaTrader 8 platform.

[mgjul]

Thanks Ray but in terms of loading time I'm sorry to contradict you but in my case I just ran 2 tests within 5min from each other to calculate exactly how long before all my charts and workspaces are operational.
With NT7 my 6 workspaces (total of 35 charts) loaded in 2 min 5 sec.
With NT8 using exactly the same number of charts/workspaces, on the same computer, with the same wifi connection (also same number of indicators, none of them from 3rd party vendors, same live datafeed from Interactive Brokers) it took exactly 24 min 33 sec. Talk about a difference.
So, I'm sure you will understand that under those circumstances I have no other choice but to stick to NT7 (which I like working with) wandering what sort of "improvements" NT8 brought to the trading world.

[junkone]

Ray,
Here is what i look for before i migrate my NT7 scripts to NT8. i monitor the board daily for the following
1. Number of crash reports. This number must become ZERO for atleast a week. i am hoping that you will find some Catch all and recover methodology. i cannot afford to put real money in a product that i dont know will be able to run throughout the trading session. this is even more important to me as i have to migrate my running strategies from NT7 to NT8 and the risk vs opportunity between these 2 is skewed in favour of NT7 until the stability issue is resolved.
2. Resource management. i have used the microsoft instrumentation in past and done some stats but would love to see performance metrics as part of the product that i can enable and disable on demand. I often use amibroker as a charting tool and find that the stats calculation that is shown on demand is very very useful for me to optimize my scripts. i am sure that as a programmer i can instrument my metrics but there a lot of processing that occurs internally in NT that i will not be privy to or have access to. but as a trader i want my overall platform running faster than the ticks/events that it recieves from the market and so the overall metrics is a very key component of the trading platform.
Cheers
Suresh

[NinjaTrader_Ray]

Thanks Ray but in terms of loading time I'm sorry to contradict you but in my case I just ran 2 tests within 5min from each other to calculate exactly how long before all my charts and workspaces are operational.
With NT7 my 6 workspaces (total of 35 charts) loaded in 2 min 5 sec.
With NT8 using exactly the same number of charts/workspaces, on the same computer, with the same wifi connection (also same number of indicators, none of them from 3rd party vendors, same live datafeed from Interactive Brokers) it took exactly 24 min 33 sec. Talk about a difference.
So, I'm sure you will understand that under those circumstances I have no other choice but to stick to NT7 (which I like working with) wandering what sort of "improvements" NT8 brought to the trading world.

Absolutely, huge difference. I am speculating that it has something to do with IB's pacing violations, we have different handling in NT8 than NT7. I have asked someone on my team more intimate with these details to look into this further. You would not see this behavior with another provider like Continuum or Kinetick, just isolated to IB. Thanks for providing this critical piece of information.

[marcel-nl]

With respect to the slow loading times, we would be interested to learn more as NinjaTrader 8 as a whole delivers much better performance than NinjaTrader 7.

This is a known issue with NT8 and Interactive Brokers. This has been raised many times during the beta. Also there is an known issue with data spikes and NT8 which did not exist in NT7.

[Eric M.]

Hello Ray,

I'm also getting similar results as mgjul - No problems with loading I.B. data into NT7 but I have a
workspace with numerous charts on a multi-monitor setup that can take up to half an hour to load with NT8 - This becomes a major problem in the event of a "Broker disconnect" and everything has to be loaded again.
On my computer I have bandwidth monitor software installed to track network traffic.
With NT7, network traffic is basically constant until all charts are loaded.
Unfortunately with NT8, I have noticed that the data cycles - peaking at around 200kbps and then falling away to around 20kbps - This continues until all the charts are loaded.
I get similar results when loading a basic chart with no indicators.

The 'Pacing Violation' issue was rarely a problem for me with NT7.
I'm hoping the data loading issue can be resolved in NT8 - This is a major negative for me !!
Apart from this I think you and your team have done a marvellous job of NT8 and I thank you for that.

Regards,
Eric

[NinjaTrader_Ray]

This is a known issue with NT8 and Interactive Brokers. This has been raised many times during the beta. Also there is an known issue with data spikes and NT8 which did not exist in NT7.

Yes, it's known difference in behavior that has unacceptable adverse effects in some situations. We are considering different options. With respect to the spike, you are correct, we are aware of this.

[futuretrader8]

Thanks Ray but in terms of loading time I'm sorry to contradict you but in my case I just ran 2 tests within 5min from each other to calculate exactly how long before all my charts and workspaces are operational.
With NT7 my 6 workspaces (total of 35 charts) loaded in 2 min 5 sec.
With NT8 using exactly the same number of charts/workspaces, on the same computer, with the same wifi connection (also same number of indicators, none of them from 3rd party vendors, same live datafeed from Interactive Brokers) it took exactly 24 min 33 sec. Talk about a difference.
So, I'm sure you will understand that under those circumstances I have no other choice but to stick to NT7 (which I like working with) wandering what sort of "improvements" NT8 brought to the trading world.

i have yet to fully test NT8 but with current NT7 my initial loading time for 20 charts is 12min, so i guess with NT8 it could hit as long as several hours for just initial loading, that basically renders NT8 useless. comparing with NT8's multiple-thread vs NT7' single thread design, the efficiency should be vastly improved, but the reality is just opposite. is that a design failure or somthing else? this is fatal for a trading platform.

[NinjaTrader_Ray]

i have yet to fully test NT8 but with current NT7 my initial loading time for 20 charts is 12min, so i guess with NT8 it could hit as long as several hours for just initial loading, that basically renders NT8 useless. comparing with NT8's multiple-thread vs NT7' single thread design, the efficiency should be vastly improved, but the reality is just opposite. is that a design failure or somthing else? this is fatal for a trading platform.

The long load time issue is specific to Interactive Brokers only. They have a historical data loading limitation called "pacing violations" which we opted to implement differently than NinjaTrader 7 resulting in long load times in some user specific scenarios. We are looking at an alternate implementation. If your twelve minute load time is through a different provider then you should expect to see faster load times.

[futuretrader8]

unfortuantely i'm trading with IB with purchased single broker IB NT license.

[NinjaTrader_Ray]

unfortuantely i'm trading with IB with purchased single broker IB NT license.

We will improve this more than likely with our next release sometime in December.

[marcel-nl]

Yes, it's known difference in behavior that has unacceptable adverse effects in some situations. We are considering different options. With respect to the spike, you are correct, we are aware of this.

Is Ninja Trader also considering options to solve the issue with data spikes and Interactive Brokers. I returned to Ninja Trader 7 pending a solution for both the slow data load and the data spikes. I see advantages wit NT8, but currently it is not workable for me.

[NinjaTrader_Brandon]

Hello marcel-nl,
Thanks for your reply.
Yes, NinjaTrader is looking into options to resolve the data spikes with Interactive Brokers in NinjaTrader 8.