Stop NT8 saving live data as historic data

[b16_aln]

Hello,

I am trying to set up Nt8 so that it doesn't save any live data as historic data at all, so that when I start NT8 I am always either pulling historic data from my data provider or using my verified and imported history files. I want to do this to ensure that there is no data contamination. Currently this is not working as expected

In terms of market data settings, everything is unchecked except, "get data from server" and "Show tick replay".

Before every NT8 re-start, I delete the My Documents>NT8> Cache and My Documents>Nt8>db>cache.

Despite this, NT8 is still saving live data to the historic data folders and when I start NT8 there is historic data saved from the previous session(s).When I go to the historic folder now for example, My Documents>Nt8>db>Tick>EURUSD, I can see NCD files in there for last weeks data, with a modified data of last week. So these files were written to the historic folder last week and have not updated or been removed.

Please advise why this is happening and how I can stop NT8 saving live data as historic.

[NinjaTrader_Jason]

Hello b16_aln,

Disabling the 'Record live data as historical' setting at Control Center-->Tools-->Options-->Market data ensures realtime data you receive is not saved as historical data in NinjaTrader. I assume you have this option disabled.

NinjaTrader loads historical data as per the information at the link below.

https://ninjatrader.com/support/helpGuides/nt8/?loading_historical_data.htm

Sorry I don't see how the EURUSD example shows that live (realtime) data is saved as historical data. Is this not historical data that was loaded from the connected broker/data feed connection last week?

As long as 'Record live data as historical' is disabled, the historical data in NinjaTrader consists of historical data that is already present in NinjaTrader or historical data that is loaded from the connected broker/data feed connection.

[b16_aln]

Yes, I have the option to record live data disabled.

The example shows that ncd data is being saved as historic on Nt8 when I have all the options set so that I don't save any historic data. I'm not sure how else I can describe it to you, if there is an NCD file then NT8 has saved data as historic data. I don;t want data saved as historic unless I validate it and import it myself. When I turn the chart on, it should be populated by data fro my broker only, not data stored in NT8 ncd files. If I leave NT8 running for a week on live data, that data gets saved as historic in NT8. If I don;t want data saved as historic then next time I turn on Nt8 there should be no ncd files on NT8. The presence of ncd files in the db folders demonstrates that the data that was presviously used was stored on NT8 as historic. Next time I use NT8 it will load whatever historic data it has saved first and then guess what other data it needs so attempts to fill the gaps. If Nt8 is set to NOT save historic data then why is it saving ncd (historic data) files?

[NinjaTrader_Jason]

When NinjaTrader loads historical data from the connected data feed/broker connection, it is stored in NinjaTrader as .ncd files. This does not indicate that live (realtime) data is being stored as historical data.

If you don't want to load historical data from the connected data feed/broker connection, please disable 'Get data from server' at Control Center-->Tools-->Options-->Market data. It will then only load data in charts that is already present in NinjaTrader.

If you manually import historical data in NinjaTrader, it will be saved as .ncd files in the db-->tick, minute and day folders as well. So if NinjaTrader only contains manually imported historical data, you will still see .ncd files in the db folder.

[b16_aln]

ok thanks, so you cannot view historical data on a chart from the data provider without NT8 saving it as historical? e.g If log on to Nt8 1 hour into the London session at 9am, NT8 will save 1hour only of historic data then the rest of the day will not be stored as historic data? So when I log out I will have a random 1 hour of data saved in historic files?

At what stage does data change from being live to being historic, take today for example, I logged on at 8am uk time, and my chart has been getting realtime data all day. Why do I have around 15 ncd historical data files saved for today starting from when I signed in this morning up until now? It appears that at some point during the day, realtime data automatically becomes historic data by definition. Only the bar being processed is realtime, the previosu bars are all historic.

Is there no way that I can view historic data as required on daily chart from my data provider without it being saved in NT8 historic data file. That would appear to be why the data starts to get unreliable because you have random patches of saved historical data sandwiched in between realtime data.

There must surely be a way of viewng real and historic data from the data provider on an as required basis without is being saved to NT8 permanent history.

[NinjaTrader_Manfred]

Hello b16_aln,

NinjaTrader does not keep all data in its memory. As real-time data arrives, it is immediately saved as historical data, which the chart then accesses when drawing/calculating charts, indicators, and other values.

To give you an example, I deleted all historical data before connecting and charting a minute-chart going back 5 days:



As you can see, NinjaTrader immediately created six files, which contain the last 5 days of minute-data for this instrument.

As this data is essential for NinjaTrader to work reliably, there is no way to keep NinjaTrader from creating these files in the first place.

The only option I see to completely avoid any saved data from being used would be to clear these folders prior to starting NinjaTrader, the platform would then be forced to redownload all historical data each morning and recreate charts based on this data alone.

Alternatively, if you right-click the chart and "Reload all historical data", any data saved locally would also be ignored. This command re-requests all historical data through the active connection and rebuilds the chart, regardless of any historical information already available.

[b16_aln]

Thank you Manfred, so just to confirm NT8 will always save live data as historical irrelevant of the settings?

I just also want to double check, when you say reload historical data will ignore data saved locally, I understood that this was not the case as per link provided and that NT8 will iterate over the data to try and determine gaps in the historical data where dowloads are required to fill missing info: https://ninjatrader.com/support/help...rical_data.htm

[NinjaTrader_Manfred]

Hello b16_aln,

The "record live data as historical" checkbox changes whether 'live' data is recorded as historical data, but not whether historical data is saved.

To explain what this means in a bit more detail; historical data is any data that already occurred and that NinjaTrader needs to download to backfill the chart. Whereas live-data is the 'real-time ticker' that arrives on the platform.

If this checkbox is disabled, NinjaTrader will request the live ticker, then use historical data (where available) to backfill the data. Once you disconnect and reconnect, it will once again request the live data ticker, then request historical data for any time missing.

E.g.:

• 10 minutes into the trading day, you connect and load an ES chart with 1 day of data
• NinjaTrader will now request 1 day of historical data, then merge this with the live ticker
• 30 minutes later, you disconnect/reconnect. NinjaTrader will now ignore the 'cached' live data from earlier and request the historical data that fills the gap going from 10 to 30 minutes, then merge it with the live ticker

That means in this case, NinjaTrader will have downloaded the 20 minutes of historical data that occurred, even though you were connected during this time, and would not use the live data it received.

If this checkbox is enabled, NinjaTrader will instead record this live data AS historical data (for this timeframe only), and then not re-request this when you reconnect.

The big difference, in this case, is not whether NinjaTrader saves the historical data, but whether the live data that arrives is turned into historical data.

As live and historical data can differ, based on the provider, a good example of how this behaves would be a long live session which is interrupted by a reconnect, at which point the chart then changes ever so slightly, as the 'old' live data is now overwritten with the provider-supplied historical data instead.

"Reload all historical data" will forego the cache and locally saved data and will request all historical data from the provider. The listed post only refers to how historical data is loaded in general. The "Reload all historical data" function, however, overrules this and does not use local data to fill the chart.

I've recorded a demonstration of how this behaves and exactly how this data is redownloaded/replaced, you may find this video at the following address:

https://www.screencast.com/t/TdIwLTj1yJVE

Please don't hesitate to let me know, should you have any further questions!

[b16_aln]

Thanks for the detailed response. My concern then is that if I have spent time to cleanse and validate all data before I manually import it as historical, every time the chart reloads, NT8 will overwrite my validated historical data with the data providers historic data?

Is there not an option to load historical data to cache or a temp folder in the same manner as the live data gets cached?

[NinjaTrader_Manfred]

Hello b16_aln,

The historical data would only be overwritten if the "Reload all historical data" is triggered. Otherwise, NinjaTrader will simply backfill anything that is missing according to the rules outlined in https://ninjatrader.com/support/help...rical_data.htm.

That means in most cases, what would happen in your case is the following:

• First, validated historical data is added to the db folder.
• As NinjaTrader now connects, it will first establish the live feed, then use the historical data you downloaded to backfill up to the point the live data begins.
• Any data missing (e.g. data between the end of your downloaded data and the beginning of the live data) would now be requested from the provider and added to this folder.
• If, after 30 minutes you reconnect, it would request these 30 minutes of historical data from the provider to replace the cached live data and add it to the existing historical data you downloaded (if recording live data as historical is enabled, it would forego this step and simply save the cached live data as historical).
• Then, it recreates the chart based on your validated historical data, mixed with the 'new' historical data filling the 'last mile', followed by the live data
• At this point, it would once again only cache the live data. As soon as you disconnect, this cached data would once again be discarded and replaced with newly downloaded historical data.

This would be the 'typical' daily behavior. The only way that "Reload all historical data" differs from how the chart draws when it is opened is that it will ignore any historical data already available and overwrite everything. That means it would first connect to live data, then backfill this live data with data from your provider, regardless of whether data already exists or not, overwriting your validated data.

As long as this is not triggered, your validated historical data would still take priority, as NinjaTrader does not replace this unless instructed.

I hope that clarifies this behavior, don't hesitate to let me know if you have any further questions!

[b16_aln]

Thanks again Manfred. Are you sure that it is the case that only when you select reload historic data manually that it is overwritten? Doesn't reload historic data get automatically fired by NT8 during a data provider reconnect following a previosu disconnect and also when you reload an indicator I thought the data was reloaded?

[NinjaTrader_Manfred]

In that case, the data would be reloaded following the guidelines in https://ninjatrader.com/support/help...rical_data.htm

That means the current day would always be loaded from the provider, while all data going back further would be taken out of the db folder (unless any data in between is missing, which will then be requested from the provider).

As re-requesting data from the provider causes delays at best (tick-data can be several MB in size per day) and missing data in the worst case, as many providers throttle historical data during market hours, NinjaTrader will always avoid pulling data from the provider unless the data it has is not sufficient.

As such, even if you reconnect or modify indicators, your existing, validated data would be preferred for any historical dates.

[Lanticalt]

Hi,

My concern is that the historical data will be mix up with NT8 data because of the future offset, the market replay download and the NT8 HDS enable connexion as connected to provider server. If I use IB as a data provider and saved 15 years of historical data into the 1 minute db files, my understanding is that the data will not be corrupt by either new necessary back fill data between the last time I close the software and the next time I will reopen it back for that same ticker?

NT8, when connecting to the provider, will fill the missing data between the historical one, the provider updated one and the one that need to be updated at the open up to the real time bar, but it does not alter the historical data in that db 1 minute file as long as I never trigger reload historical data?

It does change the tick only if the historical does not have to go that far but it will change the minute file also if the reopen was more than a minute between the last NT8 shutdown. That minutes will only be added between the existing historical data and the real time bar. If I click reload historical data, since my historical data is from the same provider, there is no problem since it would reload the same historical data in theory, (15 years would take time loading). But if the 15 years was from IQ feed for example, it would be mix up with historical data from the new connected provider IB. That 15 years of data would be erased by the new provider up to the back fill limit of that new provider, leaving the old historical data untouched for example for the last 3 years because the new have only up to 12 years of historical data.

I guess reload all historical data has to be use carefully. A pop up message explaining the consequences could be a good add up .

Now, is that how thing will happen weither record live data as historical is check or not?
And does Replay data download, Futures offset and HDS enable connexion alter the historical data by mixing kinetic or any data from NT8 server?

[NinjaTrader_Manfred]

Hello Lanticalt ,

Thank you very much for your post and welcome to the NinjaTrader forums.

The offset that applies to Future instruments would not affect the historical data itself. E.g. if there is a -7.25 offset from the ES 09-21 to the ES 03-22, this offset would simply be applied to the historical data on the chart as it is being charted.

If you disable the offset (Tools > Options > Market Data > Merge non back adjusted), the platform would still chart historical data from older expirations but would not apply the offset value.

Regarding the historical data; This would indeed be correct. Unless NinjaTrader notices it's missing historical data or is asked to reload all historical data, it would maintain these records and reuse them when it needs them. This is especially critical with providers that either have limited historical data or limit the data you may receive during market hours.

Market Replay data, on the other hand, would not be affected unless it is locally recorded. By default, this data is provided by our servers and would not differ.

Furthermore, Market Replay data is completely separate from historical data, and would not be used e.g. to backfill missing historical data.

Should you have any further questions in this regard, please don't hesitate to let me know!

[Lanticalt]

Hi,

I downloaded historical data for Forex pairs back to 2005 from IB using NT8. Every single NCD minutes file are located in the same file under for example, EUR/USD file, in which all NCD files stand from 2005 to 2022. The same apply for tick and daily files.

But when I try to do the same for futures contract I end up with many daily files in the db file itself ? More specifically, when I am trying to download futures historical data from IB servers.

First I created an ES ## ## file. But for some reason, every time I use an ES chart, the system seems to be creating those empty daily file (ES 03–10, ES 03–11, …) and leave them empty. No historical data is saved there and the db file is clutter with those daily file.

Today, I tried to download CL futures through IB Gateway. Just as the ES download did, It started to fill the db file with CL daily file (CL 03-03, CL 03–04, …) instead of creating a single file in which all minutes NCD files would be put in.

My NT8 options clearly show I have:
merge non-back adjusted check
get data from the server check
record live data as historical check so that every time I open a new chart the new data is added to the existing historical data for that contract without corrupting the previous data.

My point is, future download should be putting a single ES files in which all NCD file will end up being put so my historical data is updated properly when a chart is open later for that symbol.

Unfortunately, I am cluttered with ES, CL daily name file which are all mix up with other symbols. It is unmanageable. The day and tick files are also all clutter with daily files. Erasing them bring them back again when a chart is reopen and leave them empty.

What is wrong with futures contracts merge as non-back adjusted. Am I missing something?

QUESTIONS,

Forex, futures or any other symbols from IB server:

how do I keep updating my historical data for symbols like CL, ES, ZB, ZN, ZE, (bonds) so that a proper files is created for the NCD file to be put in for ticks, minutes, and daily file? That’s what merge non-back adjusted should be ending up doing if I am not mistaken?


Regards,

Lantic