Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Stop NT8 saving live data as historic data

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Stop NT8 saving live data as historic data

    Hello,

    I am trying to set up Nt8 so that it doesn't save any live data as historic data at all, so that when I start NT8 I am always either pulling historic data from my data provider or using my verified and imported history files. I want to do this to ensure that there is no data contamination. Currently this is not working as expected

    In terms of market data settings, everything is unchecked except, "get data from server" and "Show tick replay".

    Before every NT8 re-start, I delete the My Documents>NT8> Cache and My Documents>Nt8>db>cache.

    Despite this, NT8 is still saving live data to the historic data folders and when I start NT8 there is historic data saved from the previous session(s).When I go to the historic folder now for example, My Documents>Nt8>db>Tick>EURUSD, I can see NCD files in there for last weeks data, with a modified data of last week. So these files were written to the historic folder last week and have not updated or been removed.

    Please advise why this is happening and how I can stop NT8 saving live data as historic.

    #2
    Hello b16_aln,

    Disabling the 'Record live data as historical' setting at Control Center-->Tools-->Options-->Market data ensures realtime data you receive is not saved as historical data in NinjaTrader. I assume you have this option disabled.

    NinjaTrader loads historical data as per the information at the link below.
    https://ninjatrader.com/support/help...rical_data.htm

    Sorry I don't see how the EURUSD example shows that live (realtime) data is saved as historical data. Is this not historical data that was loaded from the connected broker/data feed connection last week?

    As long as 'Record live data as historical' is disabled, the historical data in NinjaTrader consists of historical data that is already present in NinjaTrader or historical data that is loaded from the connected broker/data feed connection.
    JasonNinjaTrader Customer Service

    Comment


      #3
      Yes, I have the option to record live data disabled.

      The example shows that ncd data is being saved as historic on Nt8 when I have all the options set so that I don't save any historic data. I'm not sure how else I can describe it to you, if there is an NCD file then NT8 has saved data as historic data. I don;t want data saved as historic unless I validate it and import it myself. When I turn the chart on, it should be populated by data fro my broker only, not data stored in NT8 ncd files. If I leave NT8 running for a week on live data, that data gets saved as historic in NT8. If I don;t want data saved as historic then next time I turn on Nt8 there should be no ncd files on NT8. The presence of ncd files in the db folders demonstrates that the data that was presviously used was stored on NT8 as historic. Next time I use NT8 it will load whatever historic data it has saved first and then guess what other data it needs so attempts to fill the gaps. If Nt8 is set to NOT save historic data then why is it saving ncd (historic data) files?

      Comment


        #4
        When NinjaTrader loads historical data from the connected data feed/broker connection, it is stored in NinjaTrader as .ncd files. This does not indicate that live (realtime) data is being stored as historical data.

        If you don't want to load historical data from the connected data feed/broker connection, please disable 'Get data from server' at Control Center-->Tools-->Options-->Market data. It will then only load data in charts that is already present in NinjaTrader.

        If you manually import historical data in NinjaTrader, it will be saved as .ncd files in the db-->tick, minute and day folders as well. So if NinjaTrader only contains manually imported historical data, you will still see .ncd files in the db folder.
        JasonNinjaTrader Customer Service

        Comment


          #5
          ok thanks, so you cannot view historical data on a chart from the data provider without NT8 saving it as historical? e.g If log on to Nt8 1 hour into the London session at 9am, NT8 will save 1hour only of historic data then the rest of the day will not be stored as historic data? So when I log out I will have a random 1 hour of data saved in historic files?

          At what stage does data change from being live to being historic, take today for example, I logged on at 8am uk time, and my chart has been getting realtime data all day. Why do I have around 15 ncd historical data files saved for today starting from when I signed in this morning up until now? It appears that at some point during the day, realtime data automatically becomes historic data by definition. Only the bar being processed is realtime, the previosu bars are all historic.

          Is there no way that I can view historic data as required on daily chart from my data provider without it being saved in NT8 historic data file. That would appear to be why the data starts to get unreliable because you have random patches of saved historical data sandwiched in between realtime data.

          There must surely be a way of viewng real and historic data from the data provider on an as required basis without is being saved to NT8 permanent history.

          Comment


            #6
            Hello b16_aln,

            NinjaTrader does not keep all data in its memory. As real-time data arrives, it is immediately saved as historical data, which the chart then accesses when drawing/calculating charts, indicators, and other values.

            To give you an example, I deleted all historical data before connecting and charting a minute-chart going back 5 days:

            Click image for larger version

Name:	historical data.png
Views:	44
Size:	98.5 KB
ID:	1138286

            As you can see, NinjaTrader immediately created six files, which contain the last 5 days of minute-data for this instrument.

            As this data is essential for NinjaTrader to work reliably, there is no way to keep NinjaTrader from creating these files in the first place.

            The only option I see to completely avoid any saved data from being used would be to clear these folders prior to starting NinjaTrader, the platform would then be forced to redownload all historical data each morning and recreate charts based on this data alone.

            Alternatively, if you right-click the chart and "Reload all historical data", any data saved locally would also be ignored. This command re-requests all historical data through the active connection and rebuilds the chart, regardless of any historical information already available.
            Manfred F.NinjaTrader Customer Service

            Comment


              #7
              Thank you Manfred, so just to confirm NT8 will always save live data as historical irrelevant of the settings?

              I just also want to double check, when you say reload historical data will ignore data saved locally, I understood that this was not the case as per link provided and that NT8 will iterate over the data to try and determine gaps in the historical data where dowloads are required to fill missing info: https://ninjatrader.com/support/help...rical_data.htm

              Comment


                #8
                Hello b16_aln,

                The "record live data as historical" checkbox changes whether 'live' data is recorded as historical data, but not whether historical data is saved.

                To explain what this means in a bit more detail; historical data is any data that already occurred and that NinjaTrader needs to download to backfill the chart. Whereas live-data is the 'real-time ticker' that arrives on the platform.

                If this checkbox is disabled, NinjaTrader will request the live ticker, then use historical data (where available) to backfill the data. Once you disconnect and reconnect, it will once again request the live data ticker, then request historical data for any time missing.

                E.g.:
                • 10 minutes into the trading day, you connect and load an ES chart with 1 day of data
                • NinjaTrader will now request 1 day of historical data, then merge this with the live ticker
                • 30 minutes later, you disconnect/reconnect. NinjaTrader will now ignore the 'cached' live data from earlier and request the historical data that fills the gap going from 10 to 30 minutes, then merge it with the live ticker
                That means in this case, NinjaTrader will have downloaded the 20 minutes of historical data that occurred, even though you were connected during this time, and would not use the live data it received.

                If this checkbox is enabled, NinjaTrader will instead record this live data AS historical data (for this timeframe only), and then not re-request this when you reconnect.

                The big difference, in this case, is not whether NinjaTrader saves the historical data, but whether the live data that arrives is turned into historical data.

                As live and historical data can differ, based on the provider, a good example of how this behaves would be a long live session which is interrupted by a reconnect, at which point the chart then changes ever so slightly, as the 'old' live data is now overwritten with the provider-supplied historical data instead.

                "Reload all historical data" will forego the cache and locally saved data and will request all historical data from the provider. The listed post only refers to how historical data is loaded in general. The "Reload all historical data" function, however, overrules this and does not use local data to fill the chart.

                I've recorded a demonstration of how this behaves and exactly how this data is redownloaded/replaced, you may find this video at the following address:

                https://www.screencast.com/t/TdIwLTj1yJVE

                Please don't hesitate to let me know, should you have any further questions!
                Manfred F.NinjaTrader Customer Service

                Comment


                  #9
                  Thanks for the detailed response. My concern then is that if I have spent time to cleanse and validate all data before I manually import it as historical, every time the chart reloads, NT8 will overwrite my validated historical data with the data providers historic data?

                  Is there not an option to load historical data to cache or a temp folder in the same manner as the live data gets cached?

                  Comment


                    #10
                    Hello b16_aln,

                    The historical data would only be overwritten if the "Reload all historical data" is triggered. Otherwise, NinjaTrader will simply backfill anything that is missing according to the rules outlined in https://ninjatrader.com/support/help...rical_data.htm.

                    That means in most cases, what would happen in your case is the following:
                    • First, validated historical data is added to the db folder.
                    • As NinjaTrader now connects, it will first establish the live feed, then use the historical data you downloaded to backfill up to the point the live data begins.
                    • Any data missing (e.g. data between the end of your downloaded data and the beginning of the live data) would now be requested from the provider and added to this folder.
                    • If, after 30 minutes you reconnect, it would request these 30 minutes of historical data from the provider to replace the cached live data and add it to the existing historical data you downloaded (if recording live data as historical is enabled, it would forego this step and simply save the cached live data as historical).
                    • Then, it recreates the chart based on your validated historical data, mixed with the 'new' historical data filling the 'last mile', followed by the live data
                    • At this point, it would once again only cache the live data. As soon as you disconnect, this cached data would once again be discarded and replaced with newly downloaded historical data.
                    This would be the 'typical' daily behavior. The only way that "Reload all historical data" differs from how the chart draws when it is opened is that it will ignore any historical data already available and overwrite everything. That means it would first connect to live data, then backfill this live data with data from your provider, regardless of whether data already exists or not, overwriting your validated data.

                    As long as this is not triggered, your validated historical data would still take priority, as NinjaTrader does not replace this unless instructed.

                    I hope that clarifies this behavior, don't hesitate to let me know if you have any further questions!
                    Manfred F.NinjaTrader Customer Service

                    Comment


                      #11
                      Thanks again Manfred. Are you sure that it is the case that only when you select reload historic data manually that it is overwritten? Doesn't reload historic data get automatically fired by NT8 during a data provider reconnect following a previosu disconnect and also when you reload an indicator I thought the data was reloaded?

                      Comment


                        #12
                        In that case, the data would be reloaded following the guidelines in https://ninjatrader.com/support/help...rical_data.htm

                        That means the current day would always be loaded from the provider, while all data going back further would be taken out of the db folder (unless any data in between is missing, which will then be requested from the provider).

                        As re-requesting data from the provider causes delays at best (tick-data can be several MB in size per day) and missing data in the worst case, as many providers throttle historical data during market hours, NinjaTrader will always avoid pulling data from the provider unless the data it has is not sufficient.

                        As such, even if you reconnect or modify indicators, your existing, validated data would be preferred for any historical dates.
                        Manfred F.NinjaTrader Customer Service

                        Comment

                        Latest Posts

                        Collapse

                        Topics Statistics Last Post
                        Started by demarcog, Today, 12:28 PM
                        1 response
                        6 views
                        0 likes
                        Last Post NinjaTrader_ChelseaB  
                        Started by demarcog, Today, 11:03 AM
                        5 responses
                        20 views
                        0 likes
                        Last Post NinjaTrader_ChelseaB  
                        Started by jo54321, Today, 04:26 AM
                        1 response
                        9 views
                        0 likes
                        Last Post NinjaTrader_ChelseaB  
                        Started by pserrera, Today, 03:26 AM
                        1 response
                        12 views
                        0 likes
                        Last Post NinjaTrader_ChelseaB  
                        Started by maxuscz, Today, 03:01 AM
                        1 response
                        9 views
                        0 likes
                        Last Post NinjaTrader_ChelseaB  
                        Working...
                        X