Please outline steps to repairing historical data gaps for chart

[adamus]

I feel I ought to know NT better than this, but I just wasted a couple of hours trying to get NT to fill in some gaps in my historical data so that my charts don't flatline.

See what I mean:

I'm connected to Interactive Brokers, but I can use IQ Feed if that doesn't work.

I tried opening the Historical Data Manager and downloading the $EURUSD data that was missing - I tried downloading both the minute data and the daily data but it didn't work.

Please would someone in Support run me through this so I have got the definitive steps 1,2,3 not just for now but because in future? Or point me to a doc or thread where it's already described? I didn't find it myself, at least not one that worked.

Thanks

[NinjaTrader_ChristopherJ]

Hello,

Thank you for your post.

Please use the following steps to resolve the issue you are experiencing:

• Right mouse click within your chart
• Data Series
• Select your Daily Data Series $EURUSD on the lower left
• Set the Session Template to <Use instrument settings>
• Select the next Data Series such as 60 Minute
• Set the Session Template to <Use instrument settings>
• Repeat this procedure for all Data Series included in this chart
• Click Ok
• Right mouse click within your chart and select "Reload all historical data"

Please let me know if the issue persists after this procedure.

[adamus]

Hi Christopher

I am using IB - I am worried that this approach will trigger the IB "NinjaTrader has reached an IB limitation for historical data requests." error.

That will mean that the download attempt just wipes the current data and is unable to download the replacement good data, surely?

[NinjaTrader_ChristopherJ]

If reloading historical data does trigger the limitation please let me know and we can use another source to load data however this process will only overwrite missing data.

[adamus]

Chris,
maybe I should have said: I have a large collection of daily, minute and tick data and I have experienced a ton of problems with data integrity with NT7.

Anyway, I checked my records to see what exactly had gone wrong with previous reloads, and it did seem that this was different.

So I tried it and it worked! Hallelulejah!

Well, I had to reduce the number of days I was loading to get past IB's data limitations, but it worked.

However I then scrolled back - I set it to load 200 days of data - and saw that in fact it had not loaded any daily data previous to Mar 2012. I then checked the daily bar chart and saw the same - there is no daily data previous to Mar 2012 on the chart despite having it set to load 1500 days.

I looked in the Historical Data Manager and saw that there is in fact a big gap in the data between Mar 2012 back to Feb 2009.

Hmmm. Where and when that gap happened, I do not know but I have a suspicion that it occured in the Historical Data Manager because I tried unsuccessfully to permanently edit a bad OHLC day from IB, but NT7 keeps overwriting my edit with the bad data. See the chart - there are in fact 2 bad days data. Sept 4th 2012 and Nov 5th 2012.

So what do I do to get the data for 2009 - 2012 back, permanently?



Thanks

[NinjaTrader_ChristopherJ]

To fill any missing Daily data please use the following steps:

• Disconnect from IB
• Connect to the Kinetick End of Day (Free) connection
• Reload data within your Daily charts to load any missing data
• Disconnect from Kinetick
• Connect to IB

Please let me know if your charts then backfill as expected.

[adamus]

Hi Chris,

my trading setup for both discretionary trading and mechanical trading is based on IB data primarily with IQ Feed as a backup, and that's what tested with and especially there is a large amount of test result data based on this configuration.

I can't just swap to Kinetick EOD data. 1 pip difference can completely change some of these results.

Obviously it has value as a work-around of last resort, but I do need to know how to maintain a historical database from 1 supplier and to have other work-arounds that let me continue to work with that supplier's data.

Thanks

[NinjaTrader_ChristopherJ]

To clarify do you want to pull real time data from IB and historical data from IQFeed? If so you would want to first disable historical data within your IB connection then connect to IB first for real time data and IQFeed second for historical data.

Keep in mind once you load historical data this will be stored within the NinjaTrader folder which will then be used for future requests and the work around would not need to be done each time.

[adamus]

That is what I have to do to deliver enough real-time data to NT7 when several strategies are running - it creates too many historical data requests if I used IB, it would cause the "pacing violation" every time I tried to restart my strategies.

What I want to do here though is to get the original IB historical data. At worst if you have no other way of doing it, I could fetch the IQ Feed data but I have until now always managed to retrieve the IB data somehow, and that's what I want to do again.

[NinjaTrader_ChristopherJ]

Unfortunately the only workaround within NinjaTrader to eliminate the pacing vioation is to request historical data from another source that does not have this limitation such as IQFeed.

[adamus]

The pacing violation is not the issue here.

I want to fill the gap for 2009 to Mar 2012 and get rid of the bad data, and avoid NT7 fetching the bad data again.

[NinjaTrader_ChristopherJ]

If IB will not provide the historical data to fill this gap you will need to load it from another source.

If you have bad data loaded you may Edit data with the steps I have provided below:

• Tools
• Historical Data Manager
• Edit
• Locate the data you would like to edit
• Double left mouse click on the value you would like to edit
• Input the new value
• Click Save
• Now within your chart please right mouse click and select "Reload all historical data".

I have provided a link below to our Help Guide that goes over Editing Data:

http://www.ninjatrader.com/support/helpGuides/nt7/editing.htm

[adamus]

Chris

before we start going round in circles, can I just point out that your first solution did not completely work.

As I said, after doing the Chart -> Reload historical data, the chart did refresh and the gaps that were originally there were gone but I found an earlier gap by scrolling to the right which I hadn't seen before.

Either NT7 messed with the data and it got deleted for reasons unknown, i.e. a bug, or the gap was always there and doing Chart -> Reload historical data did not retrieve the data for that gap.

The data reload ran to completion without giving me a pacing violation from IB, so I can only assume that IB provided all the data. However NT7 did not display it.

So what do I do? Try the Chart -> Reload Historical Data again, or use the Historical Data Manager, or what? I tried using my Daily chart for the same instrument, and it refused to download the data - IB didn't throw a pacing violation, but when the chart was repainted, the gap was still there.

Guess I should try the Historical Data Manager, right? However you guys at NT7 have to admit that something has gone awry with the data management here.

[NinjaTrader_ChristopherJ]

1. I was able to gain access to an IB test account with historical data and when testing Daily data for EURUSD data is only available from 10/2012 forward therefore when reloading data this will be the only data loaded from IB and will not fill the gap you are seeing from previous years.

2. NinjaTrader will not alter any data and will display data as sent from thru the broker/data providers API.

3. The process of loading data within the chart and within the Historical Data Manager will save data exactly the same within the appropriate data folders such as tick, minute or day.

4. The only option to fill gaps prior to 10/2012 will be to use another data source.

5. When testing with IQFeed I was able to receive data from 5/1999 forward with no gaps in data.

[adamus]

ok, thanks for working that out.

I have to decide whether to restore from backup now or use IQFeed data.

However what worries me still is the data loss. Why would NT7 wipe 3 years of data from the database? What circumstances might cause that? Do you have any bugs with the Historical Data Manager that detail this happening to other people?

Thanks