Long term review-interactive brokers effectively unusable due their data restrictions

[Calebg]

Have to post something to let everyone at Ninjatrader know, and anyone looking to do the same, how utterly frustrating it has been to try and use NT with Interactive Brokers as the sole data provider. This is mostly due to the chinsey way IB doles out data (widely documented across the internet as well as these forums), but it also has to do with NT's inability to provide some type of timeout or short circuit-function for when IB starts throttling (which is pretty much immediately). This results in me opening Ninjatrader and having it display 'Loading' for a indefinite and often permanent period of time as NT perpetually waits for data that is never going to arrive, thus rendering the entire application completely stuck.

I've spent tens of hours trying to cope with this situation and work around, and it has been very frustrating to say the least. It seems to me that NT should provide a much stronger warning about IB to customers or even boot IB entirely as a provider until and unless IB can provide an API that doesn't provide such a lousy experience for NT's customers. I'm mostly annoyed at InteractiveBrokers, but ultimately also very frustrated at now having plunked down my money for something that effectively does not work as intended and which will require me to spend a good amount of money more to get it there (not to mention all the time it took for me to realize all the above).

Please for the love of all that is good do not reply with, 'I just need to load a smaller amount of data'. To get IB to work effectively one would need to load no more than a week or two of data for a single chart (a completely useless amount for my, and I would guess, most people's need). Also, and very importantly - NT should not be sent into a permanently catatonic state that requires quitting once the data throttling by a data provider begins.

[NinjaTrader_PatrickG]

Thanks for the feedback. You're correct in that Interactive Brokers restricts the available historical data as outlined in their documentation here:

https://interactivebrokers.github.io...lity&gsc.tab=0

Specifically, this note:

Important: the above limitations apply to all our clients and it is not possible to overcome them. If your trading strategy's market data requirements are not met by our market data services please consider contacting a specialised provider.

The best solution currently would be to disconnect which would cancel the data request.

With that said, there is always room for improvement on how NinjaTrader handles API limitations. I have submitted this as a feature request to the Development Team. The internal tracking number for your feature request is SFT-3970. Please reference this internal tracking number if you ever have questions regarding this feature request.

When a feature request is implemented, you'll find it in the release notes:

http://ninjatrader.com/support/helpGuides/nt8/en-us/?release_notes.htm

[Calebg]

HI Patrick,

I appreciate/support the feature request submission, thanks.

Also, I'm sure that *you* realize that when you say, 'The best solution currently would be to disconnect which would cancel the data request' that any active strategies are then deactivated, and that whenever the next need for data comes up again (e.g., ticker change, change to data series settings) the user will be back to square one again (e.g., indefinitely "Loading"), but I catalog it here for anyone else reading along.

Fortunately, for now I have realized that for the sake of my sanity and productivity that the smarter thing for now was to simply upgrade to a multi-broker license so that I can use one of my other brokers, which doesn't have the same data limitations as IB, solely for my historical data needs. :-)

[NinjaTrader_PatrickG]

I can answer your questions from our perspective:

1. The purpose of acquiring a multibroker license key is to be able to connect to multiple brokers. If your goal is to receive data from one source (lets say Kinetick) and trade to only one broker, you could accomplish this with a single broker license key. In this example, Kinetick is not a broker, only a supplementary data provider, so it would not require you to obtain the multibroker key.
2. You can route orders to any connected broker when you have a multibroker license key. The only limitations would be if that broker doesn't support a certain asset class. For example, you couldn't trade futures to a TD Ameritrade account even if you were receiving futures from another connection because TD Ameritrade doesn't support futures in NinjaTrader. Here is a link with more information about our supported brokers and data providers:
   
   https://ninjatrader.com/support/help...y_provider.htm
3. You can define which connect provides data per asset class, but cannot sub-divide it further. For example, you could pull futures data from Interactive Brokers and equities data from TD Ameritrade. The link below goes into further detail about using multiple connections in NinjaTrader 8:
   
   https://ninjatrader.com/support/help...onnections.htm

[Calebg]

1) By going to a multi-broker license, you can use another source for data (say TD Ameritrade) and be ok with data pulls?

2) While still trading at IB? Or you will remove IB completely from your scenario?

3) Is NT able to run charts off Broker 1 but trade on Broker 2? Easily?

TD Ameritrade is working well for getting the data, at least for stocks (as PatrickG mentioned, no futures data/trading with TDA). No more Ninjatrader "Loading" lockups as with IB only.

So now I have two data connections on simultaneously (IB and TDA). As far as the where the data comes from vs where you can trade to - it doesn't matter. Meaning that even if data for a particular instrument is coming from TDA, I can execute my orders to my IB account.

Good luck!

7/24/2020 - UPDATE: TDA is now worthless as data provide for me, as they have changed their API to limit minute data to a measly 50 days (a fraction of what they used to). Looks like I'm going to have to get a separate data feed or find a broker other than TDA and/or InteractiveBrokers in order to pull up a normal chart in NinjaTrader.

[Calebg]

Adding this here since I started this thread (a couple years ago at this point). Long given up on thinking that interactive brokers data was worth a darn, I noticed in my internet-travels that a competing product to NT has gone so far as to openly trash Interactive Brokers data/reliability/API. It's quite an eye opening, blunt, and honest explanation from a developer about how useless/problematic IB data is. To the point that they are absolving themselves for any responsibility and/or support for those users who want to persist with the IB data. Who can blame them.

We make no apologies for the Sierra Chart users of Interactive Brokers. Interactive Brokers provides a garbage/very substandard API and we cannot provide a proper reliable interface to them.

So the simple fact is that it is garbage. We are not responsible in any way, never have been, for the problems that you will have , and you will have problems, and you will continue to have problems. We do not want to hear about them. - https://www.sierrachart.com/SupportB...206658#P206658

...One example of a problem someone blamed on us, is that when Sierra Chart is downloading historical data whether it be from TWS, or from the Sierra Chart servers, is that it disconnects from TWS.

IMHO it would behoove NT and it's users if NT followed suit.

[Cabopano]

Hello Calebg, your posts are very helpful to me. Thank you for your time to share to the community
I was wondering how did you solve your data provider issue?
Any recommendations?

[Calebg]

Cabopano, I thought I solved my data problem by using TD Ameritrade data, but then within months TD Ameritrade completely restricted the amount of historical data they would allow to the point where it is basically useless, and so no, I really don't have a good solution for NT data.

I could pay $70 month for Kinetic data and solve the problem, but so far this is too much of a bitter pill for me to swallow. For now I just use NT to load my custom algos, have a gander a them, and then use other platforms with included data to execute my trades and do day-to-day looking. I'm honestly not too thrilled to have invested my codebase, time, and money to something that essentially requires and ongoing and undisclosed subscription to be whole.

[Calebg]

FEB 2022 UPDATE: After al lot of futzing around I've finally figured out how to make IB marginally useful while using an IB data feed. I wouldn't call it an "acceptable" workaround exactly, but it's what I'm doing and at least keeps Ninjatrader locking up while waiting for data that will never come anytime you auto-connect to IB. Here's the solution:

1. Disable "Get data from server" from TOOLS->OPTIONS->MARKET DATA
This keeps Ninjatrader from auto-requesting data from IB whenever a connection is made or a new ticker is typed in, etc.

2. For any charts you want to have updated historical data for go to TOOLS->HISTORICAL DATA and choose the load tab. From this tab/screen you can choose the ticker and timeframe for which you want to load data - I highly recommend loading a relatively small amount at a time (e.g., several weeks not, months and years) and working your way up so that you don't choke the connection (because even from here Ninjatrader will get stuck in an infinite loop waiting for data if there is IB's data API throws an error). Once you start the data download don't mess with things and wait for it to do it thing (another good reason to start off with small bits first so that you don't have to wait forever in order to be sure it's actually loading the data).

With a little getting use to this method becomes fairly efficient and easy/quick to keep up with your charting needs. Of course, it would be much nicer if Ninjatrader could manage the mechanics of loading this data in the background or automatically when I type a ticker into a chart window, but for now this the best solution I've found. It would also be interesting to know/fix what it is about Ninjatrader's autoloading of data, i.e., when "Get data from server" is enabled, that offends IB's data limitations so quickly/easily, but I digress.