Historical data from IB

[village]

I am using IB for my data feed and am currently backtesting only with NT. I only seem to be able to get historical data of any sort back as far as Nov 2009 (i can get minute and daily data from that point no problems).

In the NT help , historical data overview, it lists IB as a historical data provider for a 'live account only'. Can you confirm that this is referring to the NT live account (rather than an IB live account), in other words the only way I am going to get more historical data via IB is to subscribe to NT.

I am presuming this is my problem, although if this is the case it seems strange that i can access about 400000 bars of minute data (without calling it historical data) but not 3000 daily bars of the same contract.

thanks

[NinjaTrader_Jason]

Hello village,

No, it refers to the IB account - if you connect to an IB demo or simulation account, you will not be able to load historical data from IB in NinjaTrader.

Could it be you now connect to an IB demo or simulation account and you already had historical minute data available in the NinjaTrader database (which you loaded using a different connectivity providerin NinjaTrader)?

[village]

Hi

No its not that ; my IB account is a live one with open positions, so no its not a demo account. I also cant have got the data I have from anywhere but IB. The only other connection i have used is Yahoo which i tried only yesterday (works fine for stocks, but i am after currencies and indices).

What happens is this;
say i run an optimise or backtest in strategy analyser. If i run it starting at say 1st jan 2010 it all works fine. if i run it agin and put an earlier start date , one of these three things happen (which one of the three happens is seemingly random);
1. The "loading data" dialogue box hangs forever, often with the date 7/11/2009 up. I end up having to close the whole thing down and start again.
2. i get a message from IB about historical data pacing violations and the analyser doesnt work at all
3. It appears to work fine and returns a report etc, BUT there are never any trades before nov 2009 and if i click on the 'chart' tab there are no bars before nov 2009.

(but if i load the chart for the same contract direct in IB it goes back as far as i want fine, so the data is there somewhere, just not getting accepted by NT)

I have been running around like a headless chook trying to get it to work so any suggestions welcome.

thanks
Village idiot

[NinjaTrader_Jason]

Unfortunately this is as expected. It is not possible to load historical daily forex data before November 2009 with IB (more than a year).

[village]

oh, ok then...

same with all futures and indices too it seems.

bit difficult to backtest with only a years worth of data .......

thanks anyway

[Stem1]

Is there a list of supported data feeds that gives how many years back historical data we can load into Ninja?
Thanks.

[NinjaTrader_Chip]

Hello Stem1,

Thank you for your post.

The following link lists all NinjaTrader-supported connection technologies:

http://www.ninjatrader-support.com/HelpGuideV6/helpguide.html?HistoricalData

However, for specific information of how much historical data is available you will need to contact the vendor and/or broker directly for the most up to date information.


Please let me know if you have any additional questions.

[Stem1]

Ok, will the option of using Ninja historical server still be available after the 18th OCT and will it still hold the same data as it currently does?
Thanks

[NinjaTrader_Jason]

Hello Stem1,

Unfortunately you will not be able to use the free Gain demo connection anymore after October 18th.

However if you have a live Gain account, you will be able to load historical forex data from our server. The data will be the same as now.

[village]

if there is a known restriction on the amount of historical data, or anything else for that matter, from a particular source, would it not be a good idea to include that info in the chart you posted a link for ?

something like IB, yes (live account only, one year back) would do

would have saved me several hours mucking about......

[NinjaTrader_Jason]

Hello village,

I will forward your suggestion to reflect such information in the help guide to my superiors and ask them if they can add this to the list of future considerations. Thank you for your feedback.

[abfc123]

Hi,

We have been running an comparison evaluation of NinjaTrader/Interactive Brokers FX Intraday against data from an Institutional platform. It is to be expected that there will be some differences given the nature of the product.
However, not only are there huge differences in OHLC data between live and demo accounts delivered from IB, the occurence of massive gaps in the Ninjatrader database make reliable backtesting impossible. Gaps of up to 2hours are the norm for some currencies. Even doing a repair DB via the options menu doesn't fix gaps. Is this a NT7 issue or IB? Looking at the NT logs for activity during these periods shows:- "The remote name could not be resolved 'www.ninjatrader.com' " and "An established connection was aborted by software on your host machine". So I presume they are related to how NT7 handles these exceptions.
The fact that so many gaps in data lie unreported and invisible to NT users surely must be an issue.
Also If i select 'save chart data as historical' from options menu, will this give me blanks if there is a disconnect? I realize you are trying to recommend Kinetic as a datasource but why continue offering IB as a source if you don't offer a warning as to the integrity of it? Also, does Kinetic offer intraday FX?
Until there is some way of verifying the integrity of data data loaded into the NinjaTrader database I would not trust the backtest results nor would I be comfortable running live trading when the strategy metrics required loading a significant lookback period of data to construct initial indicator values.
I am hoping that these are just teething issues with NT7 or that we are simply not employing procedures to avoid these issues.IB and NT are both quality products but if the market data is rubbish so to will be the P&L.

Thanks and regards

[NinjaTrader_Jason]

Hello abfc123,

Yes, you can expect to see differences in data when comparing different feeds/platform. Please see the link below for more information.

http://www.ninjatrader-support.com/HelpGuideV6/HowDoesNinjaTraderBuildChartBars.html

Please note that NinjaTrader acts like a radio receiver and displays the data is receives/loads. Can you please right click in the chart and select 'Reload Historical Data' - does it load the missing data?

Repairing the database will not fix data gaps.

As per the error messages in the log file, they seem to indicate loss of connection with IB. Could it be you were not connected properly at the times of the gaps? Make sure the lower left corner of the Control Center states 'Connected' in green.

Please be aware that historical tick data is not supported with IB, so if you have a tick based chart opened it cannot load historical data from IB.

If you enable 'save chart data as historical' it will save all incoming real-time chart data as historical data in your database.

Please note that IB is a supported broker as well, while Kinetick is a supported data feed only.

Yes, Kinetick offers intra day forex data.

[abfc123]

Hi Jason,

I will give Kinetic a go. Do you know what the max lookback is for historical 1minute FX data?
Regarding right clicking on the chart, that option is faded out on the right-click menu. It appears not possible from backtest charts. Is there any way to flush the history and reload ?

Thanks

[NinjaTrader_Jason]

I tested Kinetick on my end using a EURUSD minute chart and I was able to load historical data up to July 2005.

You can delete historical data via the Historical Data Manager (Control Center-->Tools). More information can be found at the following link.

http://www.ninjatrader.com/support/helpGuides/nt7/deleting.htm