Importing Historical Data Timestamp Error

[JustCallMeDan]

I have some historical data I wish to import, it has timestamps in UTC, as is often standard. This seems to be causing me some issues.

I select the appropriate options and try to import the data, and receive an error stating:


"Time stamp line 113819 is smaller/equal than timestamp of previous line"


This corresponds to the following lines:

20031026 005900;1.1776;1.1776;1.1776;1.1776;0
20031026 010000;1.1776;1.1776;1.1776;1.1776;0

Clearly, these are valid UTC times but NinjaTrader appears to be either failing to properly convert them to my local time as I understand it does when importing historical data (UK time, which at the above times should return to UTC from British Summer Time i.e. 'falls back'), or failing to account for the possible results of daylight saving.

The LOCAL timestamps should be 01:59 and then 01:00. It is entirely VALID that the RESULTING LOCAL second timestamp is 'smaller' than the first.

So why the error?

[NinjaTrader_Brett]

Hello,

Thanks for the note.

For me to look into this I need to export the file on my side.

Can I get you to post the file or just a section of the file that has the issue so I can test this on my side.

You can send it with http://filexpressit.com/main to ATTN: Brett at support at ninjatrader dot com

I look forward to assisting you further.

[JustCallMeDan]

Hi Brett,


This file has just been sent - it cuts off just after the offending line to save 100s of unnecessary megabytes!

I've tested with this file specifically and the error is replicated. I am selecting UTC format for import, and the 'error' line number differs by one depending if I choose the timestamp to represent start or end of the bar. In any case, it's pretty clear that the error is referring to the move from UTC 00:59 (GMT+BST 01:59) to UTC 01:00 (GMT 01:00)


Thanks!

[NinjaTrader_Brett]

Hello,

I do not get the error on my side. THe only error I get is this and it is only a warning:

4/21/2011 7:30:02 AM Default $EURUSD: Volume in line 0 is invalid since it is <= 0. This line and subsequent invalid volume lines have been modified and replaced by a value of 1.


Are you sure with this modified file it gets the error on your side?

If it does then we need to try to find out the difference between your PC and mine that would cause the error.

I look forward to assisting you further.

[JustCallMeDan]

Yep, I get the error with that specific file. My OS time zone is set to UTC with "Automatically adjust clock for Daylight Saving Time" enabled

[NinjaTrader_Brett]

Hello,

Ok thanks for that information. i will try the import again with your time settings and get back to you shortly with results.

[NinjaTrader_Brett]

Hello,

I get the same error when I use your time zone. Error most likely is you are not using the correct session template that the data was exported with. As if I use UTC + 3 for example I'm able to import the data in fine. So you do not have the time zone correctly set for this file it appears.

I look forward to assisting you further.

[JustCallMeDan]

Hmmm. I'm not sure I understand...

The timestamps in the file are UTC (no DST). In any timezone the UTC timestamps 20031026 005900 and 20031026 010000 are sequential, surely? I would have though that a historical data import in any timezone would have been successful?

Does the 'UTC' option in NinjaTrader not mean UTC outright, but actually mean UTC with DST corrections?

The data isn't the result of an export from NinjaTrader, but compiled from Dukascopy data - so it is not in any local DST correcting timezone, just UTC. Do I need to write a little program to convert this from UTC to London time, then call the resulting file 'UTC' in NinjaTrader to import it?

[NinjaTrader_Brett]

Hello,

It works like the below:

All times zones convert to your local time zone. However NinjaTrader does not know the time the historical data was exported in so you must tell it.

If you set the input data time zone to UTC + 2:00 and you import the file and your PC time is set to UTC. Then 2 hours is subtracted from the time values.

Where you can get into issue here is when times get pushed into the next day because there is such a large time difference.

Therefor please check the time that this data was exported in and then set this as the input time. This should then successfully import in.

I look forward to assisting you further.

[JustCallMeDan]

Hi Brett,


Thanks for your attempts to help, but I feel we are going in circles now!

The data has timestamps in UTC. This data is NOT a result of a data export from NinjaTrader. When I select UTC as the import timestamps, NinjaTrader fails to import the data, reporting the error quoted above.

If NinjaTrader can successfully import this data to other timezones, but not mine, then this appears to me to be an error with NinjaTrader, surely?

[NinjaTrader_Brett]

Hello,

I will try it with your exact settings. As long as your sure the data was exported in UTC format then this would then be a bug. Will check it with your time zone on my PC and you will hear back from me.

Thanks for hte patience.

[NinjaTrader_Brett]

Hello,

I reviewed this with my product manager.

He believe this is a bad data file. Is there another one you could try?

I look forward to assisting you further.

[KBJ]

I reviewed this with my product manager.

He believe this is a bad data file. Is there another one you could try?

Yes, it may be true that this is a bad data file.

For the $EURUSD contract on 20031026 (which was a Sunday), was the market even open in the early AM hours for that day?

[JustCallMeDan]

This isn't a futures contract, this is spot forex, as quoted by Dukascopy. As much as I understand that these MAY not have been tradable quotes where the error occurs during import (I'd have to ask Dukascopy if they were actually open for business on their trading platform at that date and time), the error message I receive indicates that potentially 6 months of data could be consistently 1 hour out, which is naturally unacceptable.

With regard to a possibly 'corrupt' data file, I chopped and altered the data to the SIMPLEST possible valid file and still reproduced the error.

First, to confirm the expected timestamp format, I exported historical data from NinjaTrader (old GAIN data) for the only DST adjusting date for which there was consistent data across the boundary, the Spring 2009 application of DST in London:

20090328 235100;1.3289;1.3289;1.3289;1.3289;2
20090329 000100;1.3289;1.3289;1.3289;1.3289;2
20090329 001100;1.3289;1.3289;1.3289;1.3289;2
20090329 001400;1.3289;1.3289;1.3289;1.3289;1
20090329 002000;1.3289;1.3289;1.3289;1.3289;2
20090329 002100;1.3289;1.3289;1.3289;1.3289;2
20090329 002200;1.3289;1.3289;1.3289;1.3289;2
20090329 002300;1.3289;1.3289;1.3289;1.3289;1
20090329 002400;1.3289;1.3289;1.3289;1.3289;1
20090329 002500;1.3289;1.3289;1.3289;1.3289;2
20090329 003100;1.3289;1.3289;1.3289;1.3289;4
20090329 003500;1.3289;1.3289;1.3289;1.3289;1
20090329 003600;1.3289;1.3289;1.3289;1.3289;3
20090329 003700;1.3289;1.3289;1.3289;1.3289;5
20090329 003800;1.3289;1.3289;1.3289;1.3289;4
20090329 003900;1.3289;1.3289;1.3289;1.3289;4
20090329 004100;1.3289;1.3289;1.3289;1.3289;4
20090329 004200;1.3289;1.3289;1.3289;1.3289;3
20090329 004300;1.3289;1.3289;1.3289;1.3289;4
20090329 004400;1.3289;1.3289;1.3289;1.3289;2
20090329 005100;1.3289;1.3289;1.3289;1.3289;2
20090329 010100;1.3289;1.3289;1.3289;1.3289;2
20090329 010900;1.3289;1.3289;1.3289;1.3289;4
20090329 011000;1.3289;1.3289;1.3289;1.3289;3
20090329 011100;1.3289;1.3289;1.3289;1.3289;1
20090329 012100;1.3289;1.3289;1.3289;1.3289;2
20090329 013100;1.3289;1.3289;1.3289;1.3289;2
20090329 014100;1.3289;1.3289;1.3289;1.3289;2
20090329 015100;1.3289;1.3289;1.3289;1.3289;2
20090329 020100;1.3289;1.3289;1.3289;1.3289;2

As you can see, the timestamps contain data in between 0100 and 0200, thus proving this exported data has had all DST effects removed (the local time 'springs forward' at 1am) - therefore (as I am in London) it is simply UTC.

So, with that in mind, here's my super short file for import (also attached):

20031026 005900;1.1776;1.1776;1.1776;1.1776;1
20031026 010000;1.1776;1.1776;1.1776;1.1776;1
20031026 010100;1.1776;1.1776;1.1776;1.1776;1

Also in UTC, and yet fails to import claiming line 1 has a timestamp smaller/equal to the prior.

Now, I cannot put it any simpler, this MUST be a bug in NinjaTrader as so far as I can tell these three lines are in the perfect expected format, regardless of data source.


Please let me know if you find this to be different, as I cannot progress with any backtesting without correctly imported historical data.

[NinjaTrader_Brett]

Hello,

Researching this with my product manager.

Thanks for the patience.