Timestamping

[Bend70]

Hi, trying to understand Timestamping please related to tick charts, I've read the help so understand timestamping can either be provided at the data source, or by NT locally on arrival

Could you confirm which of those please for the following feeds:

Rithmic
CQG into AMP (grandfathered early NT licence)
Continuum

Also, to hopefully understand the process, using ES 09-22 on a 2000 tick chart as example:
I sync my PC clock before starting NT each day, and have 'Break at EOD' ticked
I connect to my data feed
The historical bars will build with a new bar every 2000th tick since the first tick of the current day
The realtime data will continue this process going forward

So would having the bars timestamped at the data source, or by NT locally, have any affect on this? I assume I get the same bars either way?

Lastly, is it possible to understand what 'data source' means, and is there any difference between the above providers?
Picking Continuum as an example, I assume CME feeds to CQG, live data is passed though to CQG clients. CQG record the incoming stream and that becomes the historical data. Or does Ninjatrader provide the historical data from their own server, and act as an intermediate between CQG and Continuum clients?

Thanks very much!

[NinjaTrader_Emily]

Hello Bend70,

Thank you for your post.

We have a breakdown of the Data by Provider in our help guide here:

https://ninjatrader.com/support/helpGuides/nt8/data_by_provider.htm

I believe the column "Real-Time Timestamp" should answer your first question. Rithmic, CQG/AMP, and Continuum are all timestamped natively, meaning that the data is stamped by the data source and not stamped locally on arrival. Your understanding of the 2000 tick chart is correct. The realtime data should continue the process of building a new bar after each subsequent bar completes 2000 ticks. In the event of the data being stamped locally on your PC, we have an explanation of how that could affect how your charts look in real-time vs. reloading historical data. This example helps paint a picture based on what you are asking about. The example may be found here:

https://ninjatrader.com/support/helpGuides/nt8/index.html?how_bars_are_built.htm#UnderstandingWhyAChartCanLookDifferentAfterReloadingHistoricalDataFromTheServer

NinjaTrader records data on our own Historical Data Servers, and the connections you mentioned (Rithmic, CQG, and Continuum) all utilize the NinjaTrader historical data servers for loading historical data. For additional information regarding loading historical data, please see the following page from the help guide:

https://ninjatrader.com/support/helpGuides/nt8/loading_historical_data.htm

I hope this information helps to address your questions. If you have any additional questions or concerns, please feel free to post them here.

[Bend70]

Thanks for a great reply Emily, please could I check my updated understanding and ask a couple of follow ups

From the table you linked all the data feeds I'm looking (CQG, Rithmic, Continuum) are stamped natively
Tick bars are built locally so only the ticks themselves are downloaded from the data provider, not the tick bars themselves
Each tick is timestamped individually. In native data sources such as the three above, is the stamp attached at the exchange or somewhere downstream of that? And are CQG, Rithmic and Continuum all the same in this regard?
With all historical data being provided from NT servers, do any/all of CQG, Rithmic and Continuum use the same server, or different ones? ie is it possible the historical data seen in NT is dependant on which of these three feeds are the connection, or is this physically impossible if the server is common?

Thanks again

edit, one further question please if this doesn't cross your reply, with 'Break at EOD' ticked, how does NT know what 'tick zero' is? Is that stamped into the first tick of a new session from the exchange/data provider, or does NT derive it from a clock somewhere, for example the local PC clock? Could different session templates affect what 'tick zero' is? Thanks!

[NinjaTrader_Emily]

Hello Bend70,

Thanks for your reply. I'm glad to help.

You are correct that the ticks are downloaded from the data provider. There are three base data types: Tick, Minute, and Daily. There is even a grid that explains which base data type is used for building each bar type:

https://ninjatrader.com/support/helpGuides/nt8/index.html?how_bars_are_built.htm#UnderstandingTheUnderlyingBaseDataTypeRequiredForConstructingVariousChartBars

As far as the "native" timestamps and whether the timestamp is attached at the exchange or somewhere downstream, it is possible to receive different ticks depending on which data provider you are using. This can depend on whether the data provider aggregates data or not, although you may need to reach out to each data provider directly for additional information. Otherwise, I believe that the data is stamped by the exchange.

NinjaTrader has multiple historical data servers, and which server is used does not depend on the data provider. When you start up NinjaTrader, it will connect to one of the historical data servers randomly. By closing and restarting NinjaTrader, you could be connected to a different server than before. If you check the Log tab of the Control Center, there should be a line after starting up that shows "Using HDS hds-us-nt-XXX-ninjatrader.com" where the XXX shows the server number that you are connected to.

Please let us know if we may be of further assistance.

[Bend70]

Thanks Emily, great further help

Sorry, I cross-posted you with an edit for an additional question. I've put it (them!) below if you are able to answer please...

with 'Break at EOD' ticked, how does NT know what 'tick zero' is? Is that stamped into the first tick of a new session from the exchange/data provider, or does NT derive it from a clock somewhere, for example the local PC clock?

Could different session templates affect what 'tick zero' is?

Is tick zero the first tick of the current day, or the start of the data period being loaded (ie if 21 days data, is tick zero the start of the current day, or the 21st day back?)

And further to your last reply, presumably the numerous NT historic data servers act as a pool that shares common data, so a reconnect wouldn't get different data if the random reconnect got a different server? And do CQG, Rithmic and Continuum all share this pool, so all see the exact same data? Or is there some possibility for difference?

Appreciate the OCD here, but trying to understand some differences I've been seeing that don't seem to conform to my expectations
Thanks very much for your patience and consideration!

[NinjaTrader_Emily]

Hello Bend70,

Thank you for your patience.

Sorry I missed your edit! I appreciate you posting your questions again. Break at EOD is based on the Trading Hours (session template) that is selected in the data series. You are correct that a different session template could change which tick starts the new bar for the session. We have a page about Break at EOD here:

https://ninjatrader.com/support/helpGuides/nt8/break_at_eod.htm

There is also a small blurb in the descriptions of the data series parameters:

https://ninjatrader.com/support/helpGuides/nt8/index.html?working_with_price_data.htm#HowToEditDataSeriesParameters

Tick zero would be the first tick within the start of the session for the Trading Hours template selected. If you would like a breakdown of each session for the templates, please go to Control Center > Tools > Trading Hours. More details about the trading hours window may be found here:

https://ninjatrader.com/support/helpGuides/nt8/using_the_trading_hours_window.htm

The historical data servers usually have the same data available. Occasionally we may have a server (or servers) that is missing data/the data did not get recorded properly whereas other servers did record the data. In that case, restarting NinjaTrader could result in connecting to a server that does have the data available and our teams work diligently to backfill the missing data on the affected servers. Once historical data has been downloaded from the servers, it is cached in the NinjaTrader 8 folder within your local Documents. If you reconnect to a different provider or server, the already downloaded data will not be changed in the cache. NinjaTrader loads as much data as possible from the memory and cache and only requests data from the provider (historical data servers in this case) when the data has not already been saved in the memory/cache. This is what the Loading Historical Data page explains:

https://ninjatrader.com/support/helpGuides/nt8/index.html?loading_historical_data.htm#WhendoesNinjaTraderdownloadhistoricaldata

Once connected to a real-time data feed, NinjaTrader starts to cache the real-time ticks into memory. It then checks if the data to fulfill the bars request on your chart is already stored in the cache. If not, NinjaTrader will send a request to the historical data server to retrieve any data that is not already in the cache. Once the data is retrieved, the data is saved and cached for future reference.

Not all data connections available in NinjaTrader utilize the NinjaTrader HDS for historical data; it just so happens that the three providers you mentioned (CQG, Rithmic, and Continuum) do use the NinjaTrader historical data servers. In the case that a provider has its own historical data servers, any request to retrieve data that is not already cached would be sent to that provider's historical servers. If you utilize multiple connections, you have a choice to set your preferred provider for real-time and historical data based on instrument type by going to Control Center > Tools > Options > Market Data > Preferred connections - historical & Preferred connections - real-time. More information about setting preferred connections may be found here:

https://ninjatrader.com/support/helpGuides/nt8/index.html?multiple_connections.htm#SettingPreferredDataConnections

If you have a specific example of what you expect to see vs. what you are actually seeing in the platform and you would like us to analyze what might be happening, please provide additional details (screenshots, which data provider you are connected to, what instruments are affected, etc.) and we'd be happy to assist.

[Bend70]

Thank you very much Emily, your patience and completeness is greatly appreciated

Will read and process all you've provided, then if I can get the behaviour to reproduce over the coming days I will post again

Have a great evening!

[Bend70]

Hi Emily

Thanks to your help yesterday I think I've got a handle on the problem myself and others have been experiencing over the past few weeks

I did a series of NT restarts, reload data, and compare resulting bars around an execution yesterday. I have screenshots I can send if its helpful

I believe there is a split in your historical data pool that is quite significant, by my math

003, 007 and 009 provide identical historical data
001, 005, 006 and 008 match each other to around 1 second

but the first group is around 14 seconds different to the second group!

I understand that tick charts will always be slightly different, but understood this was primarily due to dropped ticks etc locally

Its interesting that the first group are absolutely identical, so presumably it is possible to provide at least historical data that would not change on reloads
The second group, maybe a second is a working tolerance?

But the 14 seconds hopefully you to agree is excessive to what we can expect from NT?

Look forward to your advice please. If you'd like me to email screenshots please let me know, but hopefully this will reproduce consistently

Thanks very much

[NinjaTrader_Emily]

Hello Bend70,

Thank you for that information.

I would like to investigate this further via email. Please send me your log and trace files. You can do this by going to the Control Center-> Help-> Email Support

Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default. There is also a spot where you may include one attachment, however, once the request has been sent and the email has been generated you will be able to reply back via email with additional attachments if needed.

Please reference the following ticket number in the subject of the email: ATTN: Emily

I appreciate your time and I look forward to resolving this item with you.

[Bend70]

Hi Emily, just to say I've sent log and trace as requested so if you don't receive them please post back here and I'll resend. I didn't get any system confirmation in my inbox yet. Thanks