HDManager collecting data from different sources has different session end

[adamus]

I just discovered after 6 months collecting forex data from 2 different sources that the data from the 2 sources have different Friday close last ticks..

The data collection timestamps are also collected in a 3rd way on my second computer.

The best point of reference for forex is the Friday evening close, since the rest of the week is 24 hour.

My computer timezone is set to London, UK.

Before Sun March 13th, on my dev machine, Friday close was recorded at 22:00 for the IB data.

Then on Fri Mar 18th and Mar 25th, the Friday close was at 21:00 since the data providers obviously use NY time and US Eastern time switched to DST.

On Mar 27th, London switched to DST as well and so Fri April 1st close was recorded at 22:00. (dev machine).

However there are problems. On my dev machine, I also collect FXCM data from DTN/IQFeed under different symbols, using the same forex session setting, i.e. EUR/USD = $EURUSD_FXCM.

All FXCM data has Friday close at 21:00, all year round.

All data, IB and FXCM, is collected on my dev box every weekend by downloading in the HDM.

From comparing the two data sources, e.g. 2011-03-11, I see the FXCM data which stops at 21:00 is equivalent to the IB data at 21:00 but the IB data continues to 22:00.

So I am missing an hour on the FXCM data on all Friday closes except those 2 Fridays in March.

There is another problem too.

On my production box, I connect to IB as my data provider and broker but with no historical data provision (disabled in config) and FXCM as my secondary data provider. I only have one set of instruments for the currencies, every instrument has 2 symbols defined, one for IB and one for FXCM.

Data is collected live because I have all my currency symbols displayed in the Market Analyzer.

In the HDM, the Friday close is recorded at 21:00 before Mar 13th.

Then on Fri Mar 18th and Mar 25th, the Friday close shows 20:00.

After London changed to DST on 27th, the Friday close became 21:00 (Apr 1st).

Yet I know that the machine was up and running and collecting data on those dates an hour after those times up to the defined forex session close, since I closed the strategies I run manually each Friday after the close.

The production box is set to London, UK timezone (Win7).

I haven't changed the Forex session config on either machine from the defaults. As such, the Friday close is set at Fri 17:00 UTC-5:00 Eastern Time.

What's going on here? Why the different times for the session end?

[NinjaTrader_RyanO]

Hi adamus,

Thank you for your post.

Can you please send an email to support[at]ninjatrader[dot]com using the Help > Mail to Support Feature and then please include your database and historical chart data selected to be sent as I would like to test this issue using your database and chart data.

Please reference this thread when sending that message.

[adamus]

I should have realised this was going to happen on my trading machine.

I pulled up the 'Send email' dialog and I wasn't sure which checkboxes to check for the things you wanted, so I checked them all and hit send.

It is now zipping huge amounts of the historical data and it looks like it will take a while yet and I expect sending an email with a mammoth attachment will not work if your email relay is standard.

The process won't abort either. Fortunately my strategies are all run on 60min bars so this might slow them down at the hour but I doubt it will cause much damage - at worst $10 slippage maybe.

However if you could spell out for me exactly which of those checkboxes I should check on that dialog for your investigation, I'll try it again once this huge zip-a-thon is done.

[NinjaTrader_RyanO]

adamus,

When sending that mail item the only two things that I will use to test this issue would be your chart data, and the database, as the rest of the items are not necessary as it relates to the historical data.

[NinjaTrader_RyanO]

Hi adamus,

As a further update to this issue in testing on my end and working with Kinetick data for FXCM the close times for each of those providers are different.

The IB forex close time is 5:00 PM EST, and based on the 7 hour time difference of the standard times that would correlate to 22:00, while with FXCM the close time on Friday is 4:00 PM EST. This would be the difference for the hour and why you see that for those two weekends is due to the time-zone discrepancies of daylight savings time.

The other difference is that IB data is time-stamped based on the local clock which would explain the shift during the time when Daylight savings was implemented in the US, but not quite within Europe and England.

[adamus]

OK Ryan, which of those checkboxes do you want me to check?

I see 5.

Choosing the wrong one results in a 2 Gig attachment. I looked in my tmp folder.

[NinjaTrader_RyanO]

Hi adamus,

Can you please see my last reply as in working on my end to test this further the issue stems from the fact that FXCM data closes at 4:00 PM EST/EDT, whereas IB will close at 5:00 PM EST/ EDT, and then the other variances are due to the time shifts, and how the time is recorded for NinjaTrader. IB data will be time-stamped locally based on when it is received, where FXCM data is time-stamped at the server.

Due to the differing nature of the time zones, and the close times and also Daylight Savings Time variances what we are seeing would be expected.

[adamus]

Hi Ryan,

thanks for your reply.

I am trying to work out what my options are. I am working on a couple of strategies using intraday data where I rely on the timestamp or the hour of the day. What you are saying means that I am going to have to code my strategy to handle daylight savings time and also what data I am using - I have 4 seperate instruments for EUR/USD, one for each data supplier.

Otherwise my hours will only be right for certain periods and data providers.

Is that correct?

[NinjaTrader_RyanO]

Hi adamus,

Based on the differences between the providers then you would need to account for the differences in the data within the strategy, but with the multiple instruments it would be a matter of slightly modifying the strategy based on the distinctions laid out previously and applying to the separate instruments.

[adamus]

Hi Ryan,

I'm still not satisfied I know what's going on with the data here.

The FXCM data I collected has 21:00 Friday close all year around.

You say FXCM ticks are "time-stamped" at the server. What do you mean? Do you mean the FXCM server using the EST timezone? In that case, NinjaTrader is converting the timestamp to my local timezone, otherwise I'd see the Friday close at 16:00 EST in my HDM.

So if NinjaTrader is applying the timezone difference, why isn't it also applying the DST difference which I see for the IB data over those two weeks in March and October?

[NinjaTrader_RyanO]

Hi adamus,

There are two ways to time-stamp the data and it will depend on the provider:
1. Done locally based on your PC time since the ticks will be provided to you without a time reference, and so when received it will be plotted based on the local time of your PC.

2. It is handled by the provider and in providing the tick information it will include a time-stamp which determines the precise time of the tick, and then it is plotted by NinjaTrader using that timestamp.

So, when FXCM data is time-stamped at 21:00 it is incorporating DST since it is being handled by FXCM's servers, and NinjaTrader is plotting based on those timestamps and offsetting based on your current local time, which varies due to the differences in when DST is implemented.

Due to the differences in the way data is provided you will want to backtest on each strategy and not mix the different providers when backtesting.

[adamus]

2. It is handled by the provider and in providing the tick information it will include a time-stamp which determines the precise time of the tick, and then it is plotted by NinjaTrader using that timestamp.

So, when FXCM data is time-stamped at 21:00 it is incorporating DST since it is being handled by FXCM's servers, and NinjaTrader is plotting based on those timestamps and offsetting based on your current local time, which varies due to the differences in when DST is implemented.

I don't see how a New York based timestamp can appear after conversion to my timezone with a 21:00 Friday close all year round.

If it took DST into consideration, then I would see an hour's difference in the 2 weeks in March when NY changed to DST before London.

Unfortunately I can't check the data right now to see if the timestamps for those 2 weeks are out by an hour compared to the IB data but as soon as my backup is finished tomorrow I'll do and check back here.

Basically I still don't see the logic in what you stated.

[NinjaTrader_RyanO]

adamus,

I'll look to a follow-up once your backup is complete, but the issue is that these are two separate data sources with different methods of handling data.

As NinjaTrader receives that data it is offset stored in the Historical Data Manager based on the times included (time-stamped) in the data and offset, and if a time is not included it will use your local time.

Since these differences are consistent based upon the mitigating factors we have been discussing you will want to incorporate that within the backtests of individual feeds of data but I can not say how these expected differences would be included within the strategies you are backtesting.

[adamus]

Ryan,

I am trying to work out what FXCM is doing. I can't see their timestamp, I can only see the resulting ticks or minute bars stored in the HDM after I download the week's data into the HDM.

There it is stored with the last bar closing at 21:00 all year round. What timezone is that? You said it was timestamped on the FXCM server, but that the FXCM server is in NY. So either NT7 has offset the timestamp when saving it, or FXCM actually send the timestamp with the London timezone.

I am trying to elucidate whether FXCM use DST offsets.

Thanks

[NinjaTrader_RyanO]

adamus,

Regarding those details of the FXCM data I can not say as that would be something FXCM would be able to clarify.

In terms of the data that is being provided to NinjaTrader we can see that the data is time-stamped from their servers, but the specifics of how they handle the DST transition and the methods for the timestamps they include in the data that NinjaTrader reads and charts would be based on the particulars of the feed itself.