Seeking advice on Data Feed Limitations

[skyholder84]

Hi there! To the admins, thanks for all the help provided so far. I find it amazing that you guys diligently reply to our queries late on weekend nights!

I need some advice about historical data requests and management. I currently use an Interactive Brokers account.

1) I have a trend-following strategy (Strategy A) which requires me to constantly scan through a set of about 50 tickers (FX pairs and mini-futures contracts) for tickers that meet a set of TA-based criteria. Each TA indicator relies on multi-timeframe confirmation over 6 different timebar intervals (5 min, 10 min, 30min, 60min, 240min, 1 day). Eg: I would need 6 different 20 period SMA values alone!

Does this mean that if I wish to recalculate all my TA indicator values at the close of every 5 minute bar, I would need to make at least**:

i) 50 tickers * 2 occurrences in 10 mins = 100 requests of 5 min bar data +
ii) 50 tickers * 1 occurrence in 10 mins = 50 requests of 10 min bar data
= a total of 150 different historical data requests every 10 minutes?

**At half hour intervals, I'd need historical bar data on a 30 minute time series as well! At the close of each day, I would in fact be making more than 300 historical bar data requests per 10 minutes! (50 tickers over 6 different time bar intervals)

If this is so, would I be running into a brick wall, given that my preferred broker (IB) limits historical data requests to 60 per 10 minutes, and I would be requesting them at 5 times the allowed limit. Is this analysis correct?

http://www.interactivebrokers.com/php/apiUsersGuide/apiguide/api/historical_data_limitations.htm

Could someone advise me on how I can tackle this problem as cheaply as possible? For example, (a) is there a way I can use 1 min bar data to calculate longer term TA indicator values? (b) Or would I need to subscribe to a data provider? (c) Or could someone recommend other brokers out there who are much more generous with historical data requests than Interactive Brokers?


2) I have a second mean-reversion based strategy (Strategy B) covering the same 50 instruments. Can strategy B access the TA values calculated in strategy A? If not, can both strategies access the same historical data file at the same time to make independent (and sadly identical) TA calculations? Or do the strategies have to make separate historical data requests? (Hope not!!)


Thanks and Kudos to you guys!

[NinjaTrader_Austin]

skyholder, once you have a chart up and running, there is no need to go back and reload historical data. With every new tick coming in, the tick will be processed and added to the end of the chart and then it becomes historical itself.

Unfortunately for what you wish do to, you would still be requesting too much data at once (50 requests when you start the strategy), so IB will not be sufficient for your data needs.

If you have two strategies that access the same data, the best solution would be to create an indicator that runs all of your calculations and then make that data public so both strategies can access it at the same time.

They can both access the same historical data without an issue.

Please let us know if you have any other questions.

[skyholder84]

skyholder, once you have a chart up and running, there is no need to go back and reload historical data. With every new tick coming in, the tick will be processed and added to the end of the chart and then it becomes historical itself.

Unfortunately for what you wish do to, you would still be requesting too much data at once (50 requests when you start the strategy), so IB will not be sufficient for your data needs.

If you have two strategies that access the same data, the best solution would be to create an indicator that runs all of your calculations and then make that data public so both strategies can access it at the same time.

They can both access the same historical data without an issue.

Please let us know if you have any other questions.

Hi Austin, thanks for the comprehensive help you've provided so far! Just a few more issues I would like to clarify:

1) You mentioned that when I have a chart up and running, I would no longer need to reload historical data requests. However, what if I am running automated strategies which don't use charts? Would the initial historical data file still update itself with live ticks for my strategy (running in the background) to use even if I don't have a chart up? It's not feasible for me to use charts, because then I'll need to have 50 different charts open!


2) At this page: http://www.interactivebrokers.com/ph...imitations.htm

It appears that I can make 60 historical data requests within a 10 minute time period. Since my strategy has only 50 instruments on it, won't I have to make only 50 historical data requests from start till end? If so, IB does seem sufficient for my needs, given that it allows 60 requests within a 10 minute period. Could you help me clarify this?

3) Also, if 50 requests exceeds the pacing violation, could I simply split my universe of 50 securities into two seperate but identical strategies (with 25 tickers each) and initiate each strategy 15 minutes apart to prevent a pacing violation? Would it work that way?

4) Lastly, is it possible for a longer-dated historical data request to be split into 2 seperate requests, or will the request be treated as null and invalid? (Ie: if i want data on a non-conventional 300 period hourly moving average which exceeds the data timespan allowed in a single request)


Many thanks for your patience and help!

[NinjaTrader_Bertrand]

skyholder84,

1. Yes, the historical data is stil maintained in NT's database.

2. That would also depend on how much data it would request, I believe for example an intraday data request for one day would count as one request, it's unfortunately hard to approximate you would need to test your scenario.

3. Correct, you could split requests, disconnect, wait for a data status change and reconnect to continue downloading data. Keep in mind however the currentday would always be requested - http://www.ninjatrader-support.com/H...ricalData.html

4. As I said above it's unfortunately hard to approximate when you would run exactly into the pacing violation. We normally suggest using a 3rd party datafeed as well for convenience and performance.

For NT7 Kinetick would be a economic and versatile choice here - www.kinetick.com