Historical Data

[crazyhorse2393]

Hi, I'm using Gain as a broker and I realize the limitations of the historical data they provide. However, some time ago I was using IB and when trying to load more than a year or two of historical data NT would seem to "time out" and not load the data.

I'm currently backtesting several swing trading ATM strategies using 60 min and 1 day time frames and would like a more comprehensive/historical view of how these strategies would have done over a 5 yr or 10 yr period. If I used another data vendor such as Barchart or esignal would I be able to load into NT 5 or 10 years worth of historical data to do that kind of back testing or would I face the "time out" scenario that I had with IB.

[NinjaTrader_RyanO]

Hi crazyhorse2393,

Thank you for your post.

This should not be an issue with data only providers like eSignal and BarChart. IB puts a limit on the amount of historical data you can request, whereas this does not occur with the likes of BarChart or eSignal.

[tradetree]

What happens if you have a workspace that has 10 charts with 100 days of historical 60min data per chart? That is quite typical. In that case NT requests 390 min bars X 100 X 10 = 390,000. bars of data. My guess is that about 80,000 bars is the IB limit. So does the workspace fail to retrieve the needed data? Do you then have to go into each chart and monkey with it to get it to request the data individually for just that chart (therefore not do a "refresh all historical data" as that would also fail)?

It seems to me as self-evident that NT has implementation issues around IB data retrieval that are quite clear:
1) Requesting 1 min bars for each 60 min bar in the chart, a factor of 60 in data requests!
2) Not serializing the requests so that the next chart data request doesn't happen until the throttle period is realized. You can blame the user for asking for too much data, but in a scenario where the user just wants to open a workspace and they can't, then it is hard to blame the user. NT should be able to fix this, especially where each individual chart does not exceed the IB throttle.

I also believe that IB should put into their API a way to request data such that IB only supplies data as quickly as they want rather than put this on the requester. This would be a "blocking request". Sounds like NT doesn't want to implement a block request anyway. If you are concerned about locking up the system, then having an "abort" box as you do for optimization back-testing would solve that.

Hi crazyhorse2393,

Thank you for your post.

This should not be an issue with data only providers like eSignal and BarChart. IB puts a limit on the amount of historical data you can request, whereas this does not occur with the likes of BarChart or eSignal.

[NinjaTrader_RyanO]

Hi tradetree,

Thank you for your post.

Regarding the IB data, I am not sure of the precise level of the data timeout, but based on the example and if that threshold is reached the data will load up to the level requested, and then following the time-out you can reload the historical data on each individual chart.

As for the current methods of implementation I see the issue, and I have forwarded these suggestions to development for inclusion on the list of enhancements.

Alternatively, you can use a data only provider to access the historical data beyond any type of limitation. A data provider such as eSignal, BarChart, or now Kinetick.

[tradetree]

Yes, but data providers are $70 to $100/mo. As a programmer myself I think this should be handled by NT as part of basic functionality.

[NinjaTrader_RyanO]

Hi tradetree,

Yes, that is correct and I have forwarded the suggestions to development for inclusion on the list of possible future enhancements, but the providers I mentioned were a method to use as an alternative as we take these suggestions and look into their possible implementation.