• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Very Slow Reload of Historical Data

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • BarzTrading
    started a topic Very Slow Reload of Historical Data

    Very Slow Reload of Historical Data

    I have loaded the full history of ES ##-## 15 minute bars from Kinetick. That takes several minutes. This seems unnecessarily long to me, but that's not my primary question.

    Then I change the interval to Day. It quickly reloads the daily data.

    Then I change back to 15 minute bars and it takes several minutes again to load the data. When I look at my system, the internet bandwidth is running at several megabits for several minutes. It's clearly going back to the server.

    Why? I was JUST viewing 15 minute bars?

    It's a Sunday. The end date on my DataSeries is yesterday, a habit so as to try to avoid any overhead from real time updates.

    Might this have anything to do with the "Use Backadjusted Data" on the Connections settings? I don't actually understand what that setting is supposed to control.



  • BarzTrading
    replied
    Thanks for the confirmation, Patrick. I can work with that.

    Leave a comment:


  • NinjaTrader_PatrickH
    replied
    Hello BarzTrading,

    Thank you for your response.

    To put things simply; The current contract with the Merge Policy set to Back Adjusted or Non-Back Adjusted is altered on the local installation and the actual continuous front month (using ##-##) will result in pulling that data directly from the provider as already merged and Back Adjusted.

    So if I am using the current front month (12-18 for example) and i use a Merge Policy that merges prior contracts together then I only need to download each individual contract needed. However, if I am requesting the data as merged and adjusted from the data provider (such as using ##-## with Kinetick) then I am requesting the entire data set again.

    With that said, I see the exact same behavior as you do when requesting a front month versus the continuous contract month from Kinetick. In addition, while I do not know the exact circumstances for the server download request for Kinetick I suspect that not many people are re-requesting the same data multiple times a day where the data is nearly over a decade just to compare download speeds for such a large data set.

    So it all breaks down to what you want in the end. Do you want the merge and adjusted (or not) data by the local installation? Then request the front month with the appropriate Merge Policy. Or do you want the merged and adjusted data from the provider directly? Then request the ##-## front month.

    Please let me know if you have any questions.

    Leave a comment:


  • BarzTrading
    replied
    Hi Chris,

    Let's ignore the issue about "the code seems to be off by one bar" for right now. It's minor and only confuses the issue. I think once your team understands how data caching behaves differently when using "ES ##-##" in the chart verses "ES 12-18" back adjusted that this issue may become irrelevant.

    Regarding the last question on whether the specific month syntax is recommended, my question may not have been clear or one of us has a different definition for what a continuous contract is.

    Let's get clear on our terms.

    A continuous contract results when data from more than one individual contract is used to create a series. Anytime a merge is done, the result is a continuous contract.
    An individual contract is a single futures contract defined by a symbol and a specific expiration month and year. There is no merging.

    If I enter "ES ##-##" in a chart I will always see a back adjusted continuous contract no matter what the merge policy is.

    If I enter "ES 12-18" what I see depends on the Merge Policy. If the merge policy is "Do not merge" then I will only see the individual contract data for ES 12-18. If the merge policy is "merge back adjusted" I will see a back adjusted continuous contract. If merge policy is "merge non back adjusted" then I will see a continuous contract where the prices have not been altered from the underlying individual contract that they were pulled from.

    So my question is not about whether to prefer continuous contracts over individual contracts. The question is whether typing the letters "ES ##-##" versus "ES 12-18" with back adjusted merge policy is preferred when you want to view a back adjusted continuous contract. In theory, these two things are exactly the same, only the symbol syntax differs. Well, and the caching behavior which I've outlined above.

    I look forward to hearing back from the product team on why caching behavior differs for the two different symbol syntaxes:
    1. ES ##-## minute bars (merge policy is ignored here). Does not use the local cache when the Start Date for the series is older than the available data OR older than 1/1/2007.
    2. ES 12-18 minute bars (merge back-adjusted). In theory exactly the same series as #1. NinjaTrader DOES use the local cache as one would expect regardless of what the Start Date for the series is set to.





    Last edited by BarzTrading; 10-22-2018, 04:34 PM.

    Leave a comment:


  • NinjaTrader_ChrisL
    replied
    Hello

    Thank you for the reply.

    Study 1.

    Can you confirm this on your side?
    I did confirm that it takes a bit longer to load data when you cross over a specific date in a given instrument with a continuous contract.

    Why impose these cutoffs?
    I will ask our product management team if this is expected.

    The code seems to be off by one bar. That is, if we make the start date equal to that of the first bar, then a reload occurs.
    What script are you inquiring on? Could you upload a script or post some snippets if this is your own script?

    Study 2.

    Why is (ES 12-18) different than (ES ##-##) regarding cache behavior?
    I will need to ask our product management team this question as well.

    Is the specific month syntax preferred or recommended? If so, why?
    The merge policy will organize the data for you automatically in NinjaTrader 8. It is a personal preference whether to use continuous contracts or not. It does seem that specific contracts load faster than continuous contracts. Note that the amount of historical data on your chart will not affect the real-time performance once loaded.

    I look forward to being of further. assistance.

    Leave a comment:


  • BarzTrading
    replied
    Thanks for your reply, Chris.

    OK, deleting the cache folder didn't change anything. But I've done several more experiments.

    I have only one workspace and it has only one chart window. Using NinjaTrader 8.


    STUDY #1 - Using ##-## Syntax (e.g. ES ##-##)


    Data

    6J ##-## (1 Minute bars)
    Oldest Available Date: 11/22/2005
    If the start date of the data series is 1/2/2007 or older, then we get this pattern. We'll call it Pattern A:
    • about 15 second delay
    • followed by 15 seconds to download ~60MB of data
    • about 45 second delay before chart displayed

    If the start date is 1/3/2007 or newer, then we get this pattern. We'll call it Pattern B:
    • 5-15 second delay and chart is displayed
      • There is no big data download, obviously using local data or cache

    ES ##-## (1 MInute bars)
    Oldest Available Date: 9/6/2005
    If start date of data series is 1/2/2007 or older, then we get Pattern A.
    If start date of data series is 1/3/2007 or newer, then we get Pattern B.

    CL ##-## (1 MInute bars)
    Oldest Available Date: 9/19/2006
    If start date of data series is 1/1/2007 or older, then we get Pattern A.
    If start date of data series is 1/2/2007 or newer, then we get Pattern B.

    ZL ##-## (1 Minute bars)
    Oldest Available Date: 4/20/2007
    If start date of data series is 4/20/2007 or older, then we get Pattern A.
    If start date of data series is 4/21/2007 or newer, then we get Pattern B.

    Conclusions

    It appears that 1/1/2007 is some kind of hard cutoff date for NinjaTrader for symbols having
    data that goes further back than 1/1/2007.

    Secondly, it appears that the oldest available bar also represents a cutoff even if that bar
    is newer than 2007.

    So, there are two types of cutoffs. If the start date is older than those cutoffs, then a full
    reload of the data is initiated and takes almost 10 times as long, even when a full reload
    is not actually needed.

    Questions:
    1. Can you confirm this on your side?
    2. Why impose these cutoffs?
    3. The code seems to be off by one bar. That is, if we make the start date equal to that of the first bar, then a reload occurs.

    Discussion: Why is this a problem?

    The problem arises when dealing with multiple symbols. One doesn't want to have to
    hard code the specific limits for every symbol to make sure that we don't trigger long
    delays with Pattern A.

    It would be nice to just apply a start date of say, 1/1/1950 and then NinjaTrader would
    be smart enough to say, "OK. We have all of the available data and the cache is fresh
    so we don't need to trigger a full download."

    Now, you might think that you could just impose a limit of say 4/1/2007, but that won't work.
    As you see above for ZL, since it doesn't go back that far, and many symbols don't, it will
    trigger a full reload of the data. In fact, some symbols only have a few years of minute data.

    Sure, I could figure out what all the limits are and be careful to only set appropriate start
    dates to ensure full reloads don't happen for my given set of symbols. But why should I have to?

    Shouldn't NinjaTrader be able to do the smart thing here? If the start date is older than the
    oldest available bar for a given symbol and all of that data is local and fresh, do not refetch
    it from the server.

    This would lead to a dramatic improvement for minute studies such as I'm doing across a wide
    spectrum of symbols going back as far as possible.


    STUDY #2 - Merge Policy Settings and Specific Month Syntax (e.g. ES 12-18)


    Data

    1) MergePolicy: Do Not Merge

    ES 12-18 (1 Minute bars)
    Oldest Available Date: 9/19/2017
    Always loads within 5-10 seconds regardless of start date. There appears to be some difference if the
    start date precedes the oldest bar, but it's hard to tell because the data downloads so quickly on
    my system. There is some small bandwidth usage for 1-2 seconds with every request.

    2) MergePolicy: Merge Non-backadjusted

    ES 12-18 (1 MInute bars)
    Oldest Available Date: 12/12/2007
    In this case, all minute data will redisplay in about 10 seconds or less regardless of the start date
    of the data series. The date can be 1/1/1950 or 2/23/2010. Appears to be pulling from the local
    cache and not hitting the server.

    3) MergePolicy: Merge back adjusted

    ES 12-18 (1 MInute bars)
    Oldest Available Date: 12/12/2007
    Reloads the chart within 15 seconds without pulling data from the server regardless of the start date.


    Conclusions


    It appears NinjaTrader and continuous contracts are designed to work better when using a specific
    expiration month (ES 12-18) rather than this syntax (ES ##-##). Caching behavior seems to be
    essentially as one would expect for the different merge policy settings.


    Questions:
    1. Why is (ES 12-18) different than (ES ##-##) regarding cache behavior?
    2. Is the specific month syntax preferred or recommended? If so, why?


    Discussion: Why this (might) still be a problem?

    For better performance it appears I could just be sure to use a merge policy of back-adjusted
    along with specific expiration months. However, this is still a problem when doing studies with
    multiple symbols. We have to list all of the specific expiration months for every symbol.

    Now, it appears one could created an Instrument List with all of the specific expiration months for
    each symbol. And then use Tools->Database Management -> Rollover to regularly advance the
    expiration months in the instrument list.

    This might work fine for my purposes, but it would be nice to have something like the ##-##
    syntax that would just always choose the front month. Well, this does appear to work except
    for the slow and unnecessary re-downloading of data.



    Thanks,
    Steve


    Leave a comment:


  • NinjaTrader_ChrisL
    replied
    Hello BarzTrading,

    Thank you for the post.

    I was able to import a continuous contact into NinjaTrader 8 and the data was loading normally.

    Please take the below steps to clear the NinjaTrader cache:
    • Shut down NinjaTrader.
    • Open the Documents > NinjaTrader 8 > db folder.
    • Delete the sub-folder named 'cache'.
    • Restart NinjaTrader and test.
    The continuous contract is no longer supported in NinjaTrader 8 by default, you can only import a historical continuous contract data from a .txt file. When you pull up an ES 12-18 chart do you get the same results? Can you create a new continuous chart with no indicators applied to it?

    The merge policy will tell NinjaTrader how to organize data between rollovers. Each setting is described on this page.

    I look forward to your reply.

    Leave a comment:

Latest Posts

Collapse

Topics Statistics Last Post
Started by The Big D, Today, 11:41 AM
1 response
3 views
0 likes
Last Post NinjaTrader_ChelseaB  
Started by markbb10, Today, 11:02 AM
1 response
3 views
0 likes
Last Post NinjaTrader_ChelseaB  
Started by hir04068, Today, 10:43 AM
2 responses
4 views
0 likes
Last Post NinjaTrader_Jesse  
Started by Johnny Santiago, Today, 04:50 AM
5 responses
22 views
0 likes
Last Post Johnny Santiago  
Started by Pete77, Today, 09:21 AM
1 response
11 views
0 likes
Last Post NinjaTrader_PatrickG  
Working...
X