Zenfire and Historical Data

[davewolfs]

Just a general question, does the Zenfire data for historical information come from the ZF data feed or is this information being fed from a Ninja Trader server?

[NinjaTrader_RyanO]

Hello davewolfs,

Thank you for your post.

Zen-Fire does not naturally support historical data. The historical data that is provided through ZenFire is recorded by NinjaTrader servers in real-time and then displayed as historical data.

[trader20]

Since the ZF feed has native timestamps that NinjaTrader uses to create chart bars in real time, (versus my local PC clock), why are the bars different when I reload them? You say the historical reload comes from NinjaTrader servers, so why aren't those servers using the same native ZF timestamps to build historical bars? If they are using the native ZF timestamps, then my reload bars should be identical to my real time bars.

[NinjaTrader_RyanO]

Hello Trader20,

Thank you for your post.

This is not necessarily true because of when the data is received, and this is dependent on your internet connection. Oftentimes the discrepancy between real-time data and historical data is changed because a tick is recorded in the bar that is being presently built, but when historical data is reloaded the tick is relocated based on the time stamp of the tick.

[trader20]

Your answer applies to datafeeds without native timestamps, since in that case the PC clock is used to build the bars in real time. But my question is about the ZF feed, which has *native* timestamps, so the internet connection, PC clock, etc is irrelavent to how bars are built in real time. I verified this by purposely setting my PC clock 20 minutes ahead of real time and observed the ZF bars being built per the ZF timestamps without regard to the pc clock, just as I have described.

So again, since native ZF timestamps are the *only* factor in how a bar is built (not internet connection, not pc clock), why are real time bars based on the native ZF timestamps not identical to historical bars based on the same exact native ZF timestamps?

[davewolfs]

Your answer applies to datafeeds without native timestamps, since in that case the PC clock is used to build the bars in real time. But my question is about the ZF feed, which has *native* timestamps, so the internet connection, PC clock, etc is irrelavent to how bars are built in real time. I verified this by purposely setting my PC clock 20 minutes ahead of real time and observed the ZF bars being built per the ZF timestamps without regard to the pc clock, just as I have described.

So again, since native ZF timestamps are the *only* factor in how a bar is built (not internet connection, not pc clock), why are real time bars based on the native ZF timestamps not identical to historical bars based on the same exact native ZF timestamps?

As of this week, ZF supports intraday historical data too. Why does NT truncate timestamps on ticks?

[NinjaTrader_Ray]

Your answer applies to datafeeds without native timestamps, since in that case the PC clock is used to build the bars in real time. But my question is about the ZF feed, which has *native* timestamps, so the internet connection, PC clock, etc is irrelavent to how bars are built in real time. I verified this by purposely setting my PC clock 20 minutes ahead of real time and observed the ZF bars being built per the ZF timestamps without regard to the pc clock, just as I have described.

So again, since native ZF timestamps are the *only* factor in how a bar is built (not internet connection, not pc clock), why are real time bars based on the native ZF timestamps not identical to historical bars based on the same exact native ZF timestamps?

The historical server is based off of NT 6.0 which agreggates tick data into minute bars based on 00:00:01 through 00:00:00. We changed this in NT 6.5 client so minute bars are created from ticks using 00:00:00 through 00:00:59. Thus, if you re-load data you can get subtle differences since there are two difference minute bar compression schemes used. Once we roll to NT7, the historical data server will roll to version 7 also thus the minute bars will be the same.

There was a discussion on this a month or so ago in this forum, you could do a search on it if you want further information.

[NinjaTrader_Ray]

As of this week, ZF supports intraday historical data too. Why does NT truncate timestamps on ticks?

For clarification, NT drops the millisecond precision on time stamps from Zen-Fire. The reason is that historically, there was NO time stamp coming from Zen-Fire and no other feed we supports provides granularity down to the millisecond. Supporting millisecond granularity has implications and the demand has not been overwhelming thus far. We do have this on our list for future enhancement but it will not be part of NT7.

[trader20]

OK, thank you for the explaination about how the historical data server aggregates minutes bars. That answers my question. I am glad you are fixing that in NT7. The real time and historical bars ought to be identical for datafeeds with native timestamps. I really hope that will, in fact, be the case with NT7.

[Mike_32]

Hi NT historical data is not recorded ZF data in real time. It is random generated data. I tested this long ago I was testing things with historical data enabled and then when I would restart the markers where I put where bars opened and closed the bars had changed. Then when I asked support about this they told me its not live ZF data its random generated. Since then I have historical data disabled and just record the live ZF data and it doesn't ever change. Drawback with this is if your pc is off you get no data. In NT7 are they going to have the ZF data load from a server? So if your pc goes off when you start it, it will load the data? Like Metatrader?

[NinjaTrader_Bertrand]

Mike_32, I'm not sure where you got the information from - the historical data provided is recorded ZenFire data, for the reasons outlined in the previous posts in this thread and those explained in the following link reloads from the server could produce slightly different results - http://www.ninjatrader-support.com/H...ChartBars.html

[DaManJ]

For clarification, NT drops the millisecond precision on time stamps from Zen-Fire. The reason is that historically, there was NO time stamp coming from Zen-Fire and no other feed we supports provides granularity down to the millisecond. Supporting millisecond granularity has implications and the demand has not been overwhelming thus far. We do have this on our list for future enhancement but it will not be part of NT7.

Hi Ray, I think millisecond granularity would be a most excellent feature.
You mention there are implication to this.
I can think of a few things:
Timestamp in historical db needs to be stored in a data-type which can support milliseconds (in sql server I use bigInt to store timevalues).

some APIs may or may not support milliscond - definitely Zen-fire does but maybe not some others. But does that matter?

I suppose in your c# code there would be no issue as System.DateTime supports miliseconds.

With the new database architecture in NT7 have you actually definited the datatimes in a field which can support milliseconds in futures (i.e. bigInt datatype)?
I'm just thinking if you rollout the new db with a datatype that can't handle milliseconds then it might be really hard to upgrade in future.

Would it actually matter if some providers didn't actually send you a timestamp in miliseconds? Can you just read the 1 second granularity as 1000 miliseconds?

Would be really great to have this feature. I would say above anything else, this would be #1 on my list of enhancements to make

Regards,
DaManJ

[NinjaTrader_RyanO]

Hi DaMan J,

Thank you for your suggestion.

I will forward your suggestion to development for inclusion on the list of possible future enhancements to NinjaTrader. Thank you.