Discrepancy between my historical spreadsheet data and charts (IQ-Feed on both)

[Johnny]

Hello,

While testing some of my quantitative studies of market data today and checking with my charts, I noticed a discrepancy between my historical data and my charts. Since both use the same data feed, I found that rather odd.

To clarify, I use QCollector to download historical 1-minute intraday data as a *.csv file in Excel. My data provider is IQ-Feed. This is a delayed feed. Only recently did I set this up with Ninja and have been using Continuum as my data feed in NT prior to that.

I use the ES futures contract at CME. For my historical download, I use the continuous contract, @ES# and for my charts I used the front month (currently 06-16).

The discrepancy is the gap between the Open price on the 10th of March 2016 and the Close price on the prior day, the 9th of March 2016. Basically, I have the same Open price for the 10th on both charts and my Excel file, but the Close on the 9th is different.

My Excel sheet has a Close on the 9th at 1988, while my chart has a Close at 1979.

I attach pictures of both my charts and spreadsheets to illustrate.

It seems like this is the case with several other older dates as well, but if I look at today and other recent days, my charts and historical data align perfectly.

Any ideas? I tried talking to IQ-Feed, but they haven't been able to give a good answer on this.

Have I set up my system wrong somehow?

Thanks very much in advance for any help on this! I'm really confused right now.

[Johnny]

I should add that I only look at the US Equities RTH template in Ninjafeed and use the same for my historical data.

That is 09:30-16:00 NY time.

Please don't pay any attention to the time on my charts, as it's Norwegian clock hours. 15:31 = 09:31 and 22:00 = 16:00.

[NinjaTrader_Brandon]

Hello Jonny,
Thanks for your post.
By default your charts will merge back and adjust data so that there is no gaps during rollover.
You can change this setting under Tools> Options> Data> MergePolicy
You can set this to MergeNonBackAdjsuted if you want to see the actual prices at that time.
Below I have provided a link to the understanding merge policies section of the help guide.

http://ninjatrader.com/support/helpGuides/nt7/?historical_data.htm

Additionally, rather than exporting IQFeed data then importing it into NinjaTrader, I would recommend just connecting to IQFeed in NinjaTrader.
The following link will take you to a page that details the steps required for creating a connection to IQFeed.

http://www.ninjatrader.com/ConnectionGuides/IQFeed-Connection-Guide

[Johnny]

Hello, Brandon,

Thank you very much!

You can set this to MergeNonBackAdjsuted if you want to see the actual prices at that time.

This worked like a charm, I think. For the mentioned day, my historical *.csv data from the @ES# now matches my charts in NinjaTrader.

To explain further my position, I attempt to analyze price changes and patterns from one day to the next. Or at most, perhaps the relationship between say 3-4 days. I go back in time varying from the last month/week and perhaps several years (3-5).

So, what's important for me then, is that my historical data accurately captures this relationship so to speak, i.e., difference between two sessions and specifically the actual gap between open and close.

Prior to doing the adjustment above, my historical @ES# had a gap down Open, while my charts showed a gap up Open. This is obviously a huge difference that would impact my analysis severly.

Do you think that using the @ES# for historical data and then MergeNonBackAdjusted on Ninja is the right choice for what it is I attempt to do?

Thanks very much in advance.

Johnny

[NinjaTrader_Brandon]

Hello Johnny,
Thanks for your reply.
Yes, this will work.
You could also leave the merge policy at default, connect to IQFeed, and open an ES 06-16 chart or a ES ##-## chart (continuous chart)

[Johnny]

Hello Johnny,
Thanks for your reply.
Yes, this will work.
You could also leave the merge policy at default, connect to IQFeed, and open an ES 06-16 chart.

That's what I currently do? Now using your suggestion of MergeNonBackAdjusted?

...or a ES ##-## chart (continuous chart)

Using default merge policy?

This would show exactly the same as in my data source in Excel (continuous symbol @ES#), I suppose?

Thanks.

[NinjaTrader_Brandon]

Hello Johnny,
I forgot that you do not want a continuous contract, you want data merged and not adjusted. You would want to leave the merge policy as MergeNonBackAdjusted and use the current front month.

[Johnny]

Hello Johnny,
I forgot that you do not want a continuous contract, you want data merged and not adjusted. You would want to leave the merge policy as MergeNonBackAdjusted and use the current front month.

Yes.

But alternatively, like I think you suggested, I could use the default merge policy, BUT use the continuous ES ##-##?

[Johnny]

Also, I need to get to the bottom of this for my historical analysis, but would the continuous contract be accurate for comparing days "side-by-side" like I've alluded to?

The important thing for me is that Opening Gaps based on Prior Day Close and Current Day Close is accurate.

Or that the Current Day High & Low is correct Relative to the Prior Day.


Will the continuous contract be sufficient for this? Or would I have to actually get INDEX data for this to be correct?

I would greatly appreciate it if you have any useful input or help here.

[NinjaTrader_Brandon]

Hello Johnny,
Thanks for your reply.
No, this cannot be done. Continuous contracts will be displayed in the format the data provider supplies it. Merge policies will not affect Kinetick continuous contract data.

[NinjaTrader_Brandon]

Hello Johnny,
Thanks for your reply.
If you want to view past contracts with the current front month and you want to see the actual prices you would use the current front month and MergeNonBackAdjusted. This will leave a gap at rollover from the price difference between the contacts.
If you want to view the current front month with past contracts and not have the gap at rollover you would use MergeBackAdjusted or a continuous contract. This will not give you the actual price for past contracts as they are adjusted to remove the gap.

[Johnny]

Hello again,

I once again find myself confused.

Using your suggested setting, Fronth month + MergeNonBackAdjusted, did align my charts with the continuous @ES# downloaded into my Excel sheets, i.e., a gap down open on the 10th of March 2016.

But I then figured I might just chart the @ES# directly and have the same data on my charts as my Excel.

This would be the ES ##-##, right?

However, using the ES ##-##, I once again have a gap UP on that day.

Can you explain this? Or am I using the wrong symbol for continuous IQ Feed?

Please see attached pictures. Thanks!

Johnny

[NinjaTrader_Brandon]

Hello Johnny,
Thanks for your reply.
I see that your chart is plotting rollover at the incorrect time. This is what will need to be resolve on your system.
It is expected however that with MergeNonBackadjusted that you will see a gap on the 10th since this is when the ES rolled over. Since the prices between the 03-16 contract and 06-16 contract are different, there is a gap.
I would recommend disconnecting and going to Tools> Options> Data> Reset Instruments
Exit NinjaTrader and open Documents\NinjaTrader 7\db
Delete the 'cache' folder
Restart NinjaTrader
Right-click on your chart and select Data Series
Under Session Template select Use Instrument Settings
Select OK
Right-click on your chart and select Reload All Historical Data

[Johnny]

Hello, Brandon,

I think perhaps you're confused with regards to time since it's my Norwegian computer clock at the bottom. I mentioned this initially.

So, on my chart, you have 15:30 = 09:30 NY time and 22:00 = 16:00 NY time.

I use the US equities RTH session template because I basically focus on the stock market session.

Anyway, I did what you suggested, but it didn't help.

Is perhaps the ES ##-## on Ninja the @ES#C continuous back adjusted contract? And that's why I get a discrepancy between charts and my data?

I use the @ES# continuous for download into my database, but this is not backadjusted.

Regards,

Johannes

[NinjaTrader_Brandon]

Hello Johnny,
Thanks for your reply.
I did take your time zone into consideration. It was the session template you were using that affected this in comparison to my chart.
Yes, the ES ##-## is a continuous contract that is back adjusted.
I do not know what you mean by you are using @ES#C to get data into NinjaTrader. This is not an instrument or symbol mapping in NinjaTrader. You should only be accessing data in NinjaTrader as the ES ##-## or the ES with the front month such as ES 06-16