Bug - Replay Data Time Zones

[reach4thelasers]

Steps to reproduce:

1. set up your PC so it thinks you're in London (Greenwich Mean Time) - or any time zone.
2. download replay data for an instrument for a specific day
3. Start the replay connection and play the data

Expected result:
- data starts playing from midnight 00:00 to 23:59

Actual result:
- data plays from 05:00 to 04:59

being just 5 hours ahead of EST I can live with this, but if i was in shanghai and requested replay data for 15/10/2015, I would actually get data from noon on 15/10/2015 to noon on 16/10/2015.

Who knows what I would get if I was in New Zealand - although if Ninjatrader could provide replay data for 'tomorrow' then that would be a game changer feature.

[reach4thelasers]

This might be the cause of another bug I experience without fail. If I rewind the replay connection to the start it freezes - every time. I got to close Ninjatrader and restart every time I need to rewind... Is it possibly it's trying to rewind to 00:00 and causing a failure

[strategesis]

Their market replay data universally changes from one date to the next at midnight US Eastern Time.

What it should do is change as of the first instant of the new day's session, as defined by the exchange on which the instrument trades.

[reach4thelasers]

Well the Forex Market opens at 10pm Sunday evening and Closes 10pm Friday night in my Timezone. Its a 24/5 market.

If I request data for 12th October 2015 (a monday) I get replay for 0500 on the 12th until 0500 on the 13th. If I ask for Sunday it gives me 0500 Sunday to 0500 Monday. The market isn't even open at 0500 on sunday - the data doesn't start playing until 10pm.

It might just be a semantic issue whereby the term "date" on the download dialog is misleading.

[NinjaTrader_RileyS]

reach4thelasers,

Thank you for your post and feedback!

I was able to reproduce this with the Playback Connection. I had set my timezone to London which should have showed the hours on the chart as 23:00 - 22:00 for the ES instrument, but saw same times as you. I will submit this as a bug to our Product Development team and will update a unique ID.

strategesis,

Thank you for your post and feedback as well. I was unable to reproduce this.

I had adjusted the time by moving the slider on the Playback Controller back, it took a few moments, but resumed.

Could you please provide me with your workspace file so that I may test?

You can find the workspace file by going to the Documents folder -> NinjaTrader 8 -> workspaces. The name of your workspace will be the name of the file.

And, do you have any additional steps that I might have missed? How many days of replay data were you testing?

[strategesis]

Riley,

I think you must have misunderstood my comment.

For every instrument I've ever tried, the data downloaded for Market Replay has always started at midnight US Eastern time, and ended just before midnight the following day. It has nothing to do with the time-of-day slider when connected to Market Replay.

[NinjaTrader_RileyS]

Hello All,

Thank you for your posts and feedback.

I had spoken to development and I gathered more information in regards to this.

All market replay data is saved in Eastern Time (ET) which is (UTC -5 hours). Since Market Replay is downloaded by date rather than Session Template, it will go from 12:00 AM to 11:59:59 PM for that date. So if you are in London time (UTC +5 hours), it would be 5 hours later than ET, so you would see the 5:00 AM - 4:59:59 AM.

strategesis,

What version are you on? This will be can be found by going to Help -> About... (Example: 8.0.0.5).

So that I may test more, could you please supply your workspace file?

Also, are you plotting any add-ons?

I look forward to your reply.

[strategesis]

Your description of the way Market Replay data is stored and provided (when downloaded) exactly matches my experience. I'm not experiencing a bug: It's working as it's been designed. And that's the issue.

My suggestions for improvement would be as follows:

1) The timestamps on the data should be physically stored in UTC, not in US Eastern time. That simplifies the logic for converting the physical timestamp to the user's local time, or to the time zone of the exchange on which the instrument trades. The user should be given the choice of seeing the timestamps converted either to his local time zone, or to the time zone of the exchange.

That architecture also simplifies the logic for dealing with data recorded from 1 am to 2 am (US Eastern time) on the day that US Eastern Time switches from Daylight Saving Time back to Standard Time (since all times between 1 am and 2 am are repeated for that day.)

2) When a user wants the Market Replay data for a given date, he's not interested in the date as defined by his local time zone. Nor is he interested in the date as defined by US Eastern Time. What he wants is all the data from the beginning of the trading session to the end of the trading session, as defined by the exchange and by the time Zone used by the exchange to define it's operating hours and holiday rules. So, when downloading Market Replay data for a futures contract traded on an exchange in Hong Kong, he wants the data for the session as defined by that exchange, based on Hong Kong time--regardless of what his local time zone may be.

[reach4thelasers]

There's definitely a bug here. Data is missing.

I've tested this with Market Replay for EURJPY:

- Downlaod replay data for Sunday 11th October and Monday the 12th october
- replay starts at 0500 on the 11th, with no chart painting until about 10pm on the 11th.
- replay data then stops at midnight, with no more chart painting until 0500 on 12th
------ there is 5 hours of market replay data missing between 00:00 and 05:00 on the 11th.

[reach4thelasers]

Here is a screenshot of my Replay data window - I have downloaded replay data for last week from Sunday to Friday. As you can see there are gaps in the data. Seems unpredictable where the gaps are. I can live with the EST alignment but for a Forex Instrument, the data for my timezone should show 22:00 sunday to 22:00 Friday without any gaps. Seems pretty random - couple of hours missing here and there, plus some days starting a few seconds after 5am. The "day" ending at completely random times between just before 10pm and 04:59:50

[outsource]

Hey,guys,

does NT8 has the feature to replasy RTH only?

Thanks

[reach4thelasers]

And I totally agree with Strategesis that the way these dates and times are stored needs to be reviewed. As much as I love America (Great Britain's best friend in the world) - the trading day doesn't start when American's wake up, and a new trading day certainly most certainly doesn't start at midnight EST. It starts when the Aussies and Kiwis wake up. Thats the reason that the markets open at "18:00 EST" "22:00 GMT" on a Sunday night (and that each new 24h trading day is deemed to start then too) it is solely and entirely for the fact that that alarm clocks have gone off, people have got up and gone to work and its the beginning of a new day 8 or 9am in Sydney/Wellington, the day starts when they get to work and it ends when the americans go home for the day.

http://www.cmegroup.com/trading-hours.html

Personally can live with the fact that Replay data is aligned 5am - 5am EST - because its doesn't *really* impact my concept of a "day" too much. For the Kiwis/Aus/Japanese its gonna be a head**** - they're going to request data for a "day" which in their view has already happened but from Ninjatraders point of view hasn't and its completely arbitrary - even the exchanges don't consider a new day from then.

[NinjaTrader_RileyS]

Hello Folks,

Thank you all for your great feedback.

I will submit a feature request for Market Replay data to be downloaded and starting at UTC timezone, and also to have the choice to convert to the preferred timezone or to the timezone of the exchange.

reach4thelasers,

I am currently testing October 11th & 12th to check if I see any discrepancies and will update you after I have researched this further.

outsource,

No, the data is downloaded in ETH. However, the Trading hours template that you have will show the hours that you have on your chart. If you want to see all hours that the instrument provides, you can do so by right clicking on the chart and selecting 'Data Series...' and setting the option: 'Trading hours' to Default 24/7.

Once again, thank you all for your great feedback, and I will post tomorrow with further information in regards to the EURJPY.

[NinjaTrader_RileyS]

reach4thelasers,

I have tested the Playback data for October 11th & 12th and saw discrepancies. I have forwarded a message to our IT department in hopes that the data for these days are fixed over the weekend.

As for October, Sunday the 11th, I saw that the Playback connection starts well before the data begins. I will submit a feature request for this and will update you with a unique ID when I get word back from our Product Development team.

[reach4thelasers]

Hey Riley,

My feel is that this issue isn't specific to a certain instrument or time. I experience pauses in playback pretty regularly on all forex instruments - mostly between 10pm and 5am GMT