Ninjatrader Historical Data and timezones

[ATX_tdr]

Question to the community.

I have about 4+ years of intra-day data that I have downloaded via Kinetick into Ninjatrader for various instruments, for each contract.

Over the period of time this data has been downloaded, I was in the US, Central timezone. All was good, except that now I live in the US, Eastern timezone.

So, I did a little test, I downloaded some data from Kinetick, with my PC timezone set to US, Eastern (data I had already downloaded with my PC timezone set to US, Central). I verified that the data is stored in Ninjatrader based on the timezone of the PC when it is downloaded.

I don't know why, but for some reason I expected that maybe the historical database in NT would be in say, UTC, so that if you change the timezone of the PC, years of historical data are not "wrong," but this is apparently not the case.

So, I'm wondering what to do. I could just leave my PC timezone set to US Central, but that messes up other things now that I live in US Eastern. Just downloading all the data again isn't an option because Kinetick only goes back 2 years. Any other options? I was thinking I could export all the data, then import it back, but that is a lot of work.

[NinjaTrader_Kyle]

Hello,

Thank you for your post.

NinjaTrader should automatically adjust historical data to display based on the timezone of the local PC. That said, you will need to ensure that you are restarting NinjaTrader after changing the timezone setting.

[ATX_tdr]

Ok, so how does Ninjatrader "know" how much to adjust the historical data?

I did a little test. I had some data downloaded (from Kinetick real-time) with the timezone set to CST for the 6A front month contract, which has its last trade at 4:00pm CST. If I go in and look at the historical data (using the historical data manager), the last print is at 4:00pm.

Then I changed the timezone to EST and restarted Ninjatrader.

I re-downloaded the same data (from Kinetick real-time) for the 6A front month contract. If I now go in and look at the historical data (using the historical data manager), the same print that was at 4:00pm previously is now showing at 5:00pm (which is technically correct for the timezone).

So, my question is. When I open the historical data manager and inspect the data I downloaded from Kinetick, the "timestamps" I see, how are they determined? Is the data I stored internally in say UTC, but when the grid view in the historical data manager is rendered, they are adjusted to the timezone of the PC? OR Is the data being stored in the historical database with the timestamp based on the timezone the PC it was downloaded it?

[NinjaTrader_Jason]

Hello ATX_tdr,

NinjaTrader uses the time zone of your PC clock to timestamp data. If you change the time zone of your PC clock, NinjaTrader will adjust accordingly. If you are going to make changes to your PC clock, please make sure NinjaTrader is shut down.

[ATX_tdr]

Thanks for your response.

Yes, I know that I need to shutdown and restart Ninjatrader prior to making a timezone change.

My question is regarding how Ninjatrader is internally storing downloaded historical data. I do understand that within a chart, Ninjatrader is apparently able to adjust the historical data to match the timezone of the PC and based on what I'm seeing in the historical data manager, I'm wondering how it actually does that.

From what I can tell in the historical data manager, the timezone that the PC is set to determines the time assigned to a downloaded data point (minute intraday data). So, if the PC is set to the US Central timezone and I download some minute data from Kinetick and then view it in the historical data manager, I see a given time (say 4:00pm XXX data).

If I then shutdown Ninjatrader, change the PC timezone to US Eastern, then restart Ninjatrader and open the historical data manager and view the same downloaded data, it still says the same given time as when the PC was set to US Central (say 4:00pm XXX data).

So how does Ninjatrader maintain internal consistency at that point? Is the UTC offset stored in the historical database, but hidden? I doubt it, because then the historical data manager would represent that datapoint as 5:00pm XXX data automatically.

So, what this means to me is that all the previous historical data I downloaded, when my PC was in the US Central timezone, is tied to that timezone. So if I want to switch my PC to US Eastern and still keep all that historical data consistent, I would need to export it, shift it by 1 hr and then import it back?

[NinjaTrader_Jason]

Hello ATX_tdr,

Data should adjust as per your description in post #3.

"If I now go in and look at the historical data (using the historical data manager), the same print that was at 4:00pm previously is now showing at 5:00pm (which is technically correct for the timezone)."

If data is not adjusted, please right click in the chart and select 'Reload All Historical Data'? Subsequently open the Historical Data Manager-->Edit-tab and check if data is adjusted.

If not, please close the Historical Data Manager and create a brand new chart. Open and check the Historical Data Manager-->Edit-tab once more.

[ATX_tdr]

Ok, so let me make sure I understand this using a situation example.

Use case:

1. Lets say I have my PC timezone set to US, CST.
2. I start Ninjatrader, connect to Kinetick.
3. Open the Historical Data Manager and download some intraday minute data for ES
4. Open the Historical Data Manager/Edit tab and review the downloaded data. Timestamps on the data basically reflect CST.
5. Close the Historical Data Manager and shutdown Ninjatrader.
6. Reset my PC timezone to US, EST.
7. Open the Historical Data Manager/Edit tab and inspect the data downloaded in step 4.
8. Timestamps on the data basically reflect CST still.
9. Open a chart that renders the downloaded ES data
10. Close and reopen the Historical Data Manager/Edit tab and inspect the data downloaded in step 4 and all the previously downloaded data as well...all Timestamps should now reflect EST.

Is this correct? You can see this timestamping issue most readily in data downloaded on a Friday. For the ES, the last trade is at 4:15PM CST or 5:15PM EST.

So if this is correct, this means I have to open a chart on every single instrument for which I have downloaded historical intraday data...in order for all the timestamps to be updated?

How does Ninjatrader know that it needs to update the timestamps on all this data?

[NinjaTrader_ChristopherS]

Hello ATX_tdr,

When downloading data into NinajTrader, the time stamp is based on your PC clock.

If you change the time of your PC clock the data will need to be re-downloaded to update the time stamp.

NinjaTrader maintains an internal cache of your prior accessed data. When this data does not match the data on the data providers servers, NinjaTrader will download the data from the data provider.

[ATX_tdr]

Ok, so that is the answer I was trying to get...basically, if you have historical data stored in Ninjatrader that is older than what your data provider provides (say Kinetick) your basically out of luck if you need to change your PC timezone.

[NinjaTrader_ChristopherS]

Hello ATX_tdr,

Correct. The time zone used for the time stamp of the data is not part of the internal cache.

[ATX_tdr]

That's too bad, seems like a data field that should be included in the database, especially if one is to use historical data later for doing systems testing. For some contracts, I have over 4-5 years of minute historical data stored in my NT databse.

So basically, for older historical data that I can't just re-download from my data provider, I would need to export it out of NT to flat files, write a program to shift the timestamps and then re-import all the data?

[NinjaTrader_ChristopherS]

Hello ATX_tdr,

Correct.

At a minimum, and for all instruments Kinetick offers:

180 calendar days of historical Tick data (for building tick based charts, such as range and volume)
2 years of historical Minute data
10 years of historical Daily data
For more commonly traded symbols it is likely that more extensive historical data will be available, as well.

[bltdavid]

Hi ATX_Tdr,

I have some of the same questions as you. I have been writing some code to read the .ntd files and so knowing the format of the timezone of the timestamp data inside the .ntd file is important.

First, let me say that the timezone of the timestamp data inside every .ntd file is converted to and stored as the local time zone of your PC. That is, whatever timezone the PC is set to at the moment the historical data is downloaded/imported into NinjaTrader, well the timestamps of the incoming data are converted to *that* timezone and then stored into the .ntd file.

[More about "incoming data" in a moment, as this pertains to your original question.]

So, if your PC timezone is CST and you download or import historical data, the timestamps stored in the resulting .ntd file (and viewable from HDM) will be in CST local time.

Ok, now exit NT, change your timezone to EST, restart NT, goto HDM, download different data, now you will see these timestamps of this new data reflect EST timezone. But all previous data will still show CST times.

I'm pretty sure this just summarizes your experiments.

To solve your particular problem:
I think what I'd have done in your situation is to leave the PC set to CST timezone, and then export the data to text files. (Although you could do all 3 day/minute/tick, you really just need tick data, I'll explain why later). All timestamps in all data exported from HDM to text files will be converted to UTC.

So, since your .ntd files from your years of Kinetic downloads were all stored as CST (because your PC was set to CST when those .ntd files were created), you should change your timezone back to CST and export all your old Kinetic data to text files to get the timestamps in UTC format. NT will convert from CST (the PC's current timezone) to UTC when it writes to the text output file.

In other words, you should have your PC timezone set to the timezone of .ntd files when you export the .ntd files. Now, how the heck do you know which timezone your .ntd files were created in? Alas, it's up to each user to keep track of when they change the timezone of their PC (because they moved).

The reason is (I think) because NT can't convert existing .ntd files to a new timezone because it doesn't know the timezone in use at the time the .ntd file was created. But, here is the kicker: NT thinks it doesn't need to know this. NT doesn't care because it "knows" it can punt and just redownload the data, and thus rebuild .ntd files, and poof, the timezone problem is solved. New .ntd files freshly created from newly incoming data will have timestamps that reflect your PC's current timezone. So, all NT really needs to know is the timezone in place during its last invocation. On each new invocation of NT, if the current timezone is not the same as the last saved timezone, well, NT sets a flag and invalidates all the current files, so that each file is re-downloaded and re-created. (For really old data that cannot be re-downloaded, like yours, this doesn't work well). Be careful, this sequence of events is all just theory ... so caveat emptor.

Also, from NT perspective, incoming data can come from two sources. Either you import it, or it is downloaded. As far as downloaded goes, you can do explicit downloads with HDM or NT does it under the hood for you (implicit downloads) just by adding a data series to a chart.

I think internally NT builds historical bars (1) for its charts from (and only from) data stored in the .ntd files, that is, these folders,

NinjaTrader 7/db/day
NinjaTrader 7/db/minute
NinjaTrader 7/db/tick

contain subfolders on a per instrument basis, and then contain files for Ask/Bid/Last price data -- every file anywhere under these 3 folders has the extension .ntd.

(1) Realtime bars during live trading are another story and beyond the scope of this study and speculation. Since real-time bars have a different source (not from the .ntd file), this difference probably also explains why Gomi had to invent his tick recorder, although the setting to save real-time tick data as historical data (which means the .ntd files are getting updated from real-time data, the question is how often does NT update an .ntd file when this checkbox is enabled?) probably re-aligns these stories somewhat.

Also, your db/data folder is for Market Replay data, which again, is another story, and another file format different from .ntd (I think).

I presume .ntd extensions means Ninja Tick Data or NinjaTrader Data.

My whole point is to say: your .ntd files must be the same timezone as your PC or you're charts will be off.

So, you moved to EST timezone and want to change your PC to EST timezone. First, while your PC is still set to CST export your important data using HDM. These files will be created using UTC timestamps for all exports (well, Day exports don't have minute data, so UTC is immaterial to Day data).

Now exit NT, move your 3 folders in db (day, minute, and tick) somewhere else (such as D:\NT7\Backup\CST\db). Oh, and delete the db/cache folder, too.

Now while NT is still down, change the PC to EST timezone and restart NT.

When NT restarts, it will have no historical data, because the 3 db folders day/minute/tick will all be recreated but will all be empty. From this point forward, all new .ntd files created in those 3 folders will have data who's timestamps are stored as EST (aka, your PC's local timezone).

Now, goto to HDM and import your previously exported files. Make sure and set the
timezone of imported data as UTC, since NT created the exported output files as UTC times.

Btw, it looks to me all you really have to export is your Tick data, because you can check the two boxes to rebuild Minute and Day .ntd files when importing the tick files.

I don't know if any of my speculation here is accurate. I have not tested everything I said, but have been studying the .ntd file format and trying to figure out how it all works.

Because you have old data that cannot be refreshed easily by just re-downloading it, I believe you need to jump through these hoops to preserve it (CST exported to UTC) and then change PC's timezone to EST and only then import it as UTC, where it gets converted to EST automatically when stored in new .ntd files.

If NT could import data from its own .ntd files, this would solve a lot of your problems You could move off your CST .ntd files, exit NT, delete your 3 db folders, change to EST, restart NT, and then use HDM to import the CST .ntd files, telling HDM the incoming data is in CST timestamps. I presume NT cannot import from an .ntd file, but I haven't tried it. You'll probably have to do the export/import dance like I outlined.

Hope some of this makes sense.