HDManager collecting data from different sources has different session end

[adamus]

Ryan,

surely you know whether NT7 applies the timezone difference when saving the FXCM data? You said FXCM servers in NY timestamp the data, but it is held in the HDM with a 21:00 close - i.e. UK timezone.

[NinjaTrader_RyanO]

adamus,

I took it to mean that when using FXCM data that it was coming from Kinetick, and the discussion has been in relation to that data, but the forex data included with Kinetick is from FXCM and Barclays and distinct from FXCM specifically.

At this time I want to call you to further understand the issue, and so can you please send an email to support[at]ninjatrader[dot]com and please reference this thread and provide me with a number where I can contact you and we can discuss this together remotely.

[adamus]

Hi Ryan,

I have moved my NinjaTrader user workspace across to a new machine and there I downloaded the EURUSD back to beginning of March (EURUSD_TEST) and then compared that with the data I already have (EURUSD_FXCM)

It seems the problem does not occur on this new machine.

Can you check out the chart attached and tell me if that's what you see too?

Thanks

[NinjaTrader_RyanO]

Hi adamus,

In working with this issue we did determine that there was an issue with how the data was being handled locally. For the FXCM data since it is time-stamped by the servers we do so that offset occurring (a lessening of the time zone difference based on the EDT occurring two weeks before the Euro transition.

The way to be certain would be move the chart for FXCM to March 27th when DST was changed in your location and see if the time zone difference increases from the two-week difference.

[adamus]

Alright, I see what you mean..

Just to clarify this issue I'll try to be totally unambiguous about it. Here are 4 charts of the Friday / Sunday transition from Fri Mar 4, Mar 11, Mar 25 and Apr 1.

Above is my bad FXCM, below is the newly downloaded FXCM.

The first chart shows that Friday session closed at 9pm UK time (4pm NY time) as normal for winter, and it reopened at 10pm Sunday evening (5pm NY time) - no DST involved.

The second chart shows the Friday session closing at 9pm (4pm NY) for both, again as normal with no DST. On Sun Mar 13 though DST has begun in NY. However only the new data shows this. NT7 didn't apply it to the old data. The new data is good, but if I want to save the old data, I'll have to subtract an hour starting from Sun Mar 13.

And the next chart - Fri Mar 25 - still DST in NY, but not in UK. So the Friday session close at 8:00pm (3pm NY) is correct for the new data and the old data above is still wrong.

At this point, UK DST kicks in on the Sun. So the Sun session open should be at 10:00pm (5pm NY) again as it was on Sun Mar 6, but on the old data above is at 11:00pm, 1 hours behind, and on the new data it's correct.

If you then go to Fri Apr 1, you see the chart for the new data shows that the FXCM session finished at 9:00pm (4pm NY) which is correct.

The old data shows the session closing an hour too late, all data should be adjusted forward by an hour. But then miraculously, the old data starts the Sun Apr 3 session at the correct time.

If you look at tlhe last chart, I have put IB data on there too at the bottom. It corresponds correctly bar by bar with the new FXCM data but the session stays open longer at IB.

It looks like the only adjustments to be made to the old data is during this 3 week period. I'll have to check the change-over in autumn last year to see if that is equally affected, but I should still be able to save this tick data. IQFeed only provide tick data going back 30 days, so I can't just reload it from their servers.

I'd appreciate it if you could check my conclusion. It doesn't make any sense, but that is what appears to be the situation.

Thanks

[NinjaTrader_RyanO]

Hi adamus,

Thank you for the description, and based on the screenshots and what you described then those conclusions are correct. If you recall when discussing this privately we did see that there as an issue with the old data and how it was offset.

The new data versus the old data would confirm this initial understanding in that the offset for this data was not being applied as expected within NinjaTrader. There were factors contributing such as the time-stamps on data from FXCM among others discussed.

However, the offset as being applied to the current data is correct and it is with the old data where the issue resided which if I understand correctly is what these shots and descriptions are confirming.

[adamus]

Ryan,

I just confirmed with DTN/IQFeed what times they have and got this reply, for the record:

Hello Adam, as far as start and stop times for FXCM goes they are 16:00 EDT on Friday and re-opens on Sunday evening at 17:00 EDT for the new week. During the week we will create the daily bar at 17:00 EDT everyday with the exception of Friday. As to the timestamps these are provided for every single trade that comes through our system and is based on EDT or Eastern Daylight Time currently. If you have any other problems or questions please let us know.

Since the issue with NinjaTrader doesn't occur anymore, I think that wraps it up.

Thanks for the help.

[adamus]

Hi again

I am re-importing the minute data from FXCM to get rid of the Daylight Saving Time problem that we discussed here, and all is OK back to March 2009.

On 2009-03-22, the Sunday session start time is 20:00 (the week after it's 22:00 which is normal).

I will ask the data provider whether the session actually started an hour earlier on that date and earlier, but it is big co-incidence that it is right on the change-over for Daylight Saving Time.

Could you check this at your end too please? I'd appreciate it.

[NinjaTrader_RyanO]

adamus,

Using new data from FXCM for that same time period I see on March 22nd, 2009 an open time of Sunday local of 2:00 PM, which based on the Eastern Time Zone and daylight savings is offset the next week as you are seeing as well except in my local timezone, rather than the UTC+0 zone you are seeing.

This changes the following week but only by an hour, so if the time-zone is offset by an hour forward the following week it may be showing the same difference we saw earlier, but you can remove the data from the week leading up and after the transition and then reload from FXCM directly with a chart.

[adamus]

Hi Ryan

I've re-read your message a couple of times now and I'm struggling to see where that leaves me.

I basically said in my previous message that I'm seeing the incorrect timestamp handling issue again when importing data from 2009 and earlier.

You said you see the same thing - is that right?

Are you also saying that the solution is to delete the affected data from the Historical Data Manager and to reload it by plotting the affected time period on a chart for each instrument?

I've got no worries about doing that, but I'd just appreciate it if you could confirm that is what you mean, because it'll take me a while - I'm looking at over 20 instruments and 7 years of history.

Thanks

[NinjaTrader_RyanO]

Hi adamus,

My apologies if I wasn't as clear as I could have been, but in working with this issue what we are seeing is unique to your data.

In detailing the methods of FXCM data, and when the markets close, and the expected offset we see with new data pulled directly from the server the issue is arising from the imported data, and so to resolve we will want to remove the data where it is showing the offset incorrectly, and then reload on a chart.

I have tested this by removing all of my historical data and pulling the EURUSD.FXCM instrument to make sure that the offset is accurately downloaded from the provider servers during the Daylight Savings Transition in 2009 which would indicate that the time zone offset we are seeing now is related to the initial issue with the data from 2010.

One thing you may also want to do is run Windows Update as there are sometimes updates related to Daylight Savings Time offsets.

[adamus]

Hi Ryan

I am pursuing this matter with IQFeed as well and I think it might be that the FXCM data for some reason starts at 16:00 NY time on 2009-03-22. I see some random start times on Sundays earlier than that too. IQFeed stated that they are not aware of any different session times before 2010.

I didn't get a reply yet about the data.

There could be different reasons why FXCM has this data starting at 16:00 NY time on a Sunday - firstly it could be that FXCM have in some way made a Daylight Saving offset error, or secondly it might just be extra data collected from one of their sources but not cleaned up.

What makes it difficult to pin down in this case is that I have no other reliable source of data for that time, since Interactive Brokers only provide a year of past data. Do you have data you can compare it to, to see if the 5 mins chart for instance is shifted by an hour from another data providers data on that day?

Thanks

[NinjaTrader_RyanO]

adamus,

I can test IQFeed and Kinetick data as the only providers that will extend back that far within my charts. In testing both of those I see the same behavior where on 3/22 the data opens an hour prior to the open time the following week until the Daylight Savings Times correspond in the different time zones and a standard offset is in place.

This would indicate that the data is being loaded to match the local time zone offset and daylight savings time dates as recognized by my PC.

[adamus]

OK, so it's pretty weird, but nothing's going wrong, then?

[NinjaTrader_RyanO]

adamus,

Correct, as the data I am seeing directly from the server that would indicate that the data is being offset with time-stamps from the local server time to match the dates and times of Daylight Savings Time, and the offset you are seeing is related to the data itself.