• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Very Slow Reload of Historical Data

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Very Slow Reload of Historical Data

    I have loaded the full history of ES ##-## 15 minute bars from Kinetick. That takes several minutes. This seems unnecessarily long to me, but that's not my primary question.

    Then I change the interval to Day. It quickly reloads the daily data.

    Then I change back to 15 minute bars and it takes several minutes again to load the data. When I look at my system, the internet bandwidth is running at several megabits for several minutes. It's clearly going back to the server.

    Why? I was JUST viewing 15 minute bars?

    It's a Sunday. The end date on my DataSeries is yesterday, a habit so as to try to avoid any overhead from real time updates.

    Might this have anything to do with the "Use Backadjusted Data" on the Connections settings? I don't actually understand what that setting is supposed to control.



    #2
    Hello BarzTrading,

    Thank you for the post.

    I was able to import a continuous contact into NinjaTrader 8 and the data was loading normally.

    Please take the below steps to clear the NinjaTrader cache:
    • Shut down NinjaTrader.
    • Open the Documents > NinjaTrader 8 > db folder.
    • Delete the sub-folder named 'cache'.
    • Restart NinjaTrader and test.
    The continuous contract is no longer supported in NinjaTrader 8 by default, you can only import a historical continuous contract data from a .txt file. When you pull up an ES 12-18 chart do you get the same results? Can you create a new continuous chart with no indicators applied to it?

    The merge policy will tell NinjaTrader how to organize data between rollovers. Each setting is described on this page.

    I look forward to your reply.
    Chris L.NinjaTrader Customer Service

    Comment


      #3
      Thanks for your reply, Chris.

      OK, deleting the cache folder didn't change anything. But I've done several more experiments.

      I have only one workspace and it has only one chart window. Using NinjaTrader 8.


      STUDY #1 - Using ##-## Syntax (e.g. ES ##-##)


      Data

      6J ##-## (1 Minute bars)
      Oldest Available Date: 11/22/2005
      If the start date of the data series is 1/2/2007 or older, then we get this pattern. We'll call it Pattern A:
      • about 15 second delay
      • followed by 15 seconds to download ~60MB of data
      • about 45 second delay before chart displayed

      If the start date is 1/3/2007 or newer, then we get this pattern. We'll call it Pattern B:
      • 5-15 second delay and chart is displayed
        • There is no big data download, obviously using local data or cache

      ES ##-## (1 MInute bars)
      Oldest Available Date: 9/6/2005
      If start date of data series is 1/2/2007 or older, then we get Pattern A.
      If start date of data series is 1/3/2007 or newer, then we get Pattern B.

      CL ##-## (1 MInute bars)
      Oldest Available Date: 9/19/2006
      If start date of data series is 1/1/2007 or older, then we get Pattern A.
      If start date of data series is 1/2/2007 or newer, then we get Pattern B.

      ZL ##-## (1 Minute bars)
      Oldest Available Date: 4/20/2007
      If start date of data series is 4/20/2007 or older, then we get Pattern A.
      If start date of data series is 4/21/2007 or newer, then we get Pattern B.

      Conclusions

      It appears that 1/1/2007 is some kind of hard cutoff date for NinjaTrader for symbols having
      data that goes further back than 1/1/2007.

      Secondly, it appears that the oldest available bar also represents a cutoff even if that bar
      is newer than 2007.

      So, there are two types of cutoffs. If the start date is older than those cutoffs, then a full
      reload of the data is initiated and takes almost 10 times as long, even when a full reload
      is not actually needed.

      Questions:
      1. Can you confirm this on your side?
      2. Why impose these cutoffs?
      3. The code seems to be off by one bar. That is, if we make the start date equal to that of the first bar, then a reload occurs.

      Discussion: Why is this a problem?

      The problem arises when dealing with multiple symbols. One doesn't want to have to
      hard code the specific limits for every symbol to make sure that we don't trigger long
      delays with Pattern A.

      It would be nice to just apply a start date of say, 1/1/1950 and then NinjaTrader would
      be smart enough to say, "OK. We have all of the available data and the cache is fresh
      so we don't need to trigger a full download."

      Now, you might think that you could just impose a limit of say 4/1/2007, but that won't work.
      As you see above for ZL, since it doesn't go back that far, and many symbols don't, it will
      trigger a full reload of the data. In fact, some symbols only have a few years of minute data.

      Sure, I could figure out what all the limits are and be careful to only set appropriate start
      dates to ensure full reloads don't happen for my given set of symbols. But why should I have to?

      Shouldn't NinjaTrader be able to do the smart thing here? If the start date is older than the
      oldest available bar for a given symbol and all of that data is local and fresh, do not refetch
      it from the server.

      This would lead to a dramatic improvement for minute studies such as I'm doing across a wide
      spectrum of symbols going back as far as possible.


      STUDY #2 - Merge Policy Settings and Specific Month Syntax (e.g. ES 12-18)


      Data

      1) MergePolicy: Do Not Merge

      ES 12-18 (1 Minute bars)
      Oldest Available Date: 9/19/2017
      Always loads within 5-10 seconds regardless of start date. There appears to be some difference if the
      start date precedes the oldest bar, but it's hard to tell because the data downloads so quickly on
      my system. There is some small bandwidth usage for 1-2 seconds with every request.

      2) MergePolicy: Merge Non-backadjusted

      ES 12-18 (1 MInute bars)
      Oldest Available Date: 12/12/2007
      In this case, all minute data will redisplay in about 10 seconds or less regardless of the start date
      of the data series. The date can be 1/1/1950 or 2/23/2010. Appears to be pulling from the local
      cache and not hitting the server.

      3) MergePolicy: Merge back adjusted

      ES 12-18 (1 MInute bars)
      Oldest Available Date: 12/12/2007
      Reloads the chart within 15 seconds without pulling data from the server regardless of the start date.


      Conclusions


      It appears NinjaTrader and continuous contracts are designed to work better when using a specific
      expiration month (ES 12-18) rather than this syntax (ES ##-##). Caching behavior seems to be
      essentially as one would expect for the different merge policy settings.


      Questions:
      1. Why is (ES 12-18) different than (ES ##-##) regarding cache behavior?
      2. Is the specific month syntax preferred or recommended? If so, why?


      Discussion: Why this (might) still be a problem?

      For better performance it appears I could just be sure to use a merge policy of back-adjusted
      along with specific expiration months. However, this is still a problem when doing studies with
      multiple symbols. We have to list all of the specific expiration months for every symbol.

      Now, it appears one could created an Instrument List with all of the specific expiration months for
      each symbol. And then use Tools->Database Management -> Rollover to regularly advance the
      expiration months in the instrument list.

      This might work fine for my purposes, but it would be nice to have something like the ##-##
      syntax that would just always choose the front month. Well, this does appear to work except
      for the slow and unnecessary re-downloading of data.



      Thanks,
      Steve


      Comment


        #4
        Hello

        Thank you for the reply.

        Study 1.

        Can you confirm this on your side?
        I did confirm that it takes a bit longer to load data when you cross over a specific date in a given instrument with a continuous contract.

        Why impose these cutoffs?
        I will ask our product management team if this is expected.

        The code seems to be off by one bar. That is, if we make the start date equal to that of the first bar, then a reload occurs.
        What script are you inquiring on? Could you upload a script or post some snippets if this is your own script?

        Study 2.

        Why is (ES 12-18) different than (ES ##-##) regarding cache behavior?
        I will need to ask our product management team this question as well.

        Is the specific month syntax preferred or recommended? If so, why?
        The merge policy will organize the data for you automatically in NinjaTrader 8. It is a personal preference whether to use continuous contracts or not. It does seem that specific contracts load faster than continuous contracts. Note that the amount of historical data on your chart will not affect the real-time performance once loaded.

        I look forward to being of further. assistance.
        Chris L.NinjaTrader Customer Service

        Comment


          #5
          Hi Chris,

          Let's ignore the issue about "the code seems to be off by one bar" for right now. It's minor and only confuses the issue. I think once your team understands how data caching behaves differently when using "ES ##-##" in the chart verses "ES 12-18" back adjusted that this issue may become irrelevant.

          Regarding the last question on whether the specific month syntax is recommended, my question may not have been clear or one of us has a different definition for what a continuous contract is.

          Let's get clear on our terms.

          A continuous contract results when data from more than one individual contract is used to create a series. Anytime a merge is done, the result is a continuous contract.
          An individual contract is a single futures contract defined by a symbol and a specific expiration month and year. There is no merging.

          If I enter "ES ##-##" in a chart I will always see a back adjusted continuous contract no matter what the merge policy is.

          If I enter "ES 12-18" what I see depends on the Merge Policy. If the merge policy is "Do not merge" then I will only see the individual contract data for ES 12-18. If the merge policy is "merge back adjusted" I will see a back adjusted continuous contract. If merge policy is "merge non back adjusted" then I will see a continuous contract where the prices have not been altered from the underlying individual contract that they were pulled from.

          So my question is not about whether to prefer continuous contracts over individual contracts. The question is whether typing the letters "ES ##-##" versus "ES 12-18" with back adjusted merge policy is preferred when you want to view a back adjusted continuous contract. In theory, these two things are exactly the same, only the symbol syntax differs. Well, and the caching behavior which I've outlined above.

          I look forward to hearing back from the product team on why caching behavior differs for the two different symbol syntaxes:
          1. ES ##-## minute bars (merge policy is ignored here). Does not use the local cache when the Start Date for the series is older than the available data OR older than 1/1/2007.
          2. ES 12-18 minute bars (merge back-adjusted). In theory exactly the same series as #1. NinjaTrader DOES use the local cache as one would expect regardless of what the Start Date for the series is set to.





          Last edited by BarzTrading; 10-22-2018, 03:34 PM.

          Comment


            #6
            Hello BarzTrading,

            Thank you for your response.

            To put things simply; The current contract with the Merge Policy set to Back Adjusted or Non-Back Adjusted is altered on the local installation and the actual continuous front month (using ##-##) will result in pulling that data directly from the provider as already merged and Back Adjusted.

            So if I am using the current front month (12-18 for example) and i use a Merge Policy that merges prior contracts together then I only need to download each individual contract needed. However, if I am requesting the data as merged and adjusted from the data provider (such as using ##-## with Kinetick) then I am requesting the entire data set again.

            With that said, I see the exact same behavior as you do when requesting a front month versus the continuous contract month from Kinetick. In addition, while I do not know the exact circumstances for the server download request for Kinetick I suspect that not many people are re-requesting the same data multiple times a day where the data is nearly over a decade just to compare download speeds for such a large data set.

            So it all breaks down to what you want in the end. Do you want the merge and adjusted (or not) data by the local installation? Then request the front month with the appropriate Merge Policy. Or do you want the merged and adjusted data from the provider directly? Then request the ##-## front month.

            Please let me know if you have any questions.
            Patrick H.NinjaTrader Customer Service

            Comment


              #7
              Thanks for the confirmation, Patrick. I can work with that.

              Comment

              Latest Posts

              Collapse

              Topics Statistics Last Post
              Started by Erwin Beckers, Today, 01:23 AM
              0 responses
              2 views
              0 likes
              Last Post Erwin Beckers  
              Started by Mark Taylor, Yesterday, 11:36 PM
              0 responses
              3 views
              0 likes
              Last Post Mark Taylor  
              Started by Bluebeep, Yesterday, 08:38 PM
              0 responses
              7 views
              0 likes
              Last Post Bluebeep  
              Started by stokhastic, Yesterday, 07:08 PM
              0 responses
              3 views
              0 likes
              Last Post stokhastic  
              Started by jlkramer16, Yesterday, 04:29 PM
              1 response
              4 views
              0 likes
              Last Post jlkramer16  
              Working...
              X