• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Why does NT6 download new data when I already have history in the DB?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Why does NT6 download new data when I already have history in the DB?

    Nt6 has a long pause/hang when connecting to either IB or Opentick.
    I already have valid historical data, but it seems like NT wants to download and overlay this .

    Both IB and OpenTick are slow backfillers, so this action for NT is a real pain.
    And it can and does lockup NT from time to time.. especially if I get impatient and try opening another chart. This starts another backfill .. and that often locks NT up and I have to kill it via XP TaskMgr

    I've attached a MarkAnalyzer template that will help show the problem.
    Nothing special, just 3 columns with a 5//15/60 minute CCI
    And I only have about 8 instruments in it.

    Try this ..

    Open 3 charts .. say YM, ER2, and 6E (euro)
    Add this Mkt Analyzer

    Connect to Opentick ...

    IF NT survives this (50/50 for me) then disconnect and reconnect OpenTick
    You get the same wait again.
    Infact, you get wait just changing periods .. from 5 to 10 to 1 minutes for example.
    Which is bad, as the minute data is already in the DB.

    Change the symbol on a chart while it's got the 'data loading'
    for the Mkt Analyzer .. usually good for a hangup.


    I don't get any hangs if NT is not connected to a feed. It uses the historical data in the DB as expected.
    Attached Files

    #2
    imported post

    I will check to see if there is anying unusual from the set up you experienced. A few things:

    - Please make sure you are on the latest beta 9 as there were changes to internal bar data management
    - Any time you open a chart, NT will take your bars back value and check if there is historical data in the database on that day. If it is empty, it will request the entire period up to today of data. If this day falls on a Saturday where the dbase is sure to be empty, then it will always re-request data. Assuming there is data there, it will only request the current days data. This must be done since you need to have the most current data. OpenTick and IB can be extremely slow, especially OpenTick. I have seen it backfill on 15 days for over an hour, I have seen it hang. This is their current level of service. They are working hard on improving it but thats what you get for afree service.

    Ray
    RayNinjaTrader Customer Service

    Comment


      #3
      imported post

      I have attached a screenshot with steps to recreate the NT hang on backfill.

      The trick to making it hang seems to be to adjust the period of a chart .. After it has backfilled, but before NT has finished backfilling all the symbols in the mkt analyzer

      Yes ,, this is Beta 9
      Attached Files

      Comment


        #4
        imported post

        Zoltran,

        Thanks for the detailed step by step walkthrough. I can reproduce and will forward to development.

        Also, I will respond to your CCI post later on once I get clarification on a few items.

        Ray
        RayNinjaTrader Customer Service

        Comment


          #5
          imported post

          Thank you.

          Could you also inform dev't that NT is generally unstable anytime the 'Loading Data' dialogue is displayed. I've had several hangs other than the one I detailed below, just from trying to do 'things' while a backfill was proceeding elsewhere.

          By "things" .. I mean modify a chart, change period, load a template elsewhere ...
          At one time I thought it was when another dialog had to be opened .. but I can't be more specific.

          fwiw,
          Machine is XP-MediaCtr, SP2 , AMDx2, 3G Ram, tons of disk.


          Comment


            #6
            imported post

            The lock up problem when toggling chart intervals while NT is loading data will be fixed with next beta/release. Thanks for pointing that out.
            DierkNinjaTrader Customer Service

            Comment


              #7
              imported post

              Another test case for you.

              Disconnect from esignal
              Create 3 charts
              Create 3 Market Analysrs using the 3cci template posted previously
              mktAnlz 1 - 1 - default list
              mktAnlz 2 dow30
              mktAnlz 3 nasd 100

              Connect to esignal

              Wait ... Wait Wait ... it still takes a loooonnnnngggg time, even with esig

              And eventually NT will crash.

              OR at least it has the last 3 times I've tried it.

              Comment


                #8
                imported post

                I could not reproduce. Here is what I did step for step.

                - Start NT
                - Open 3 charts ES, ER2 and NQ, all 1 min with 15 days back
                - Open 3 MA's with your template
                - Default instrument list, DOW and NAZ lists
                -Connected to eSignal
                - Yes, it did take a long time to load and I will look into possibilities to improve this but after it was all loaded, it did not crash

                See anything that I may have done differently than you? Also, next time it happens, is there error message with a details button? If yes, please press and copy the text and email it over.

                Thanks

                Ray
                RayNinjaTrader Customer Service

                Comment


                  #9
                  imported post

                  I will try again later this weekend

                  Comment


                    #10
                    imported post

                    >I will try again later this weekend

                    Hi Zoltran. How did it go. I have a similar problem and if you resolved it I would like to know how. thx.

                    Antraman

                    Comment


                      #11
                      imported post

                      We did improve historical data download for Market Analyzer with Beta 10. The problem prior to Beta 10 was that there were redudant requests for download taking place.

                      Ray
                      RayNinjaTrader Customer Service

                      Comment


                        #12
                        imported post

                        Hi Ray

                        Sorry to say, NT6 Beta 10 still has some problems in this regard.

                        For example, I currently have a 5 min chart, with a Format Data Series back to early 2004. This was orginally opened using Opentick .. I wanted to see how far back I could get data.

                        This morning, I connected to Esignal .. and NT tried to backfill the whole 3 years again .. taking about 20 to 30 mins to do so.

                        I'm not sure of the logic being used.
                        Most programs I am familiar with only backfill the gaps and do not even refresh the current day. This is usually a very-good-thing, especially with provides like IB that have differing methods for r/T vs historical (.2 sec snaps vs 2 sec snaps)

                        To my thinking NT should only fill the most recent hole from the last existing quote in the DB to now. This would cover 'most' situations. And we should have a simple way to 'request' a longer backfill from the chart for those situatations the call for it.

                        Comment


                          #13
                          imported post

                          Yes, eSignal can be slow on minute bar requests.

                          The current logic backfills based on the first day of the historical data set. If there is data in the database, it will only load the current day, if there is no data, it will load the entire set again.

                          The way to resolve this particular case is to change the lookback period to fall ona day where data exists in the database. I also suggest for long lookback periods not to have the start data fall on a Saturday where there will never be historical data.

                          Ray
                          RayNinjaTrader Customer Service

                          Comment


                            #14
                            imported post

                            I"m not sure it's working as you say Ray.

                            It always seems to backfill, regardless of state or starting date.
                            For example, I just tested with charts that started last Monday.

                            Load NT, no connect - charts display fine.. the data is there
                            Start esig .. blank-out while it backfills
                            Stop esig (not NT) .. charts are fine
                            Restart esig connection ... same blank-out/backfill

                            Does it on opentic and IB as far as I can tell.
                            Of course, with IB, it's pretty certain you'll hit their throttle with NT working like this.


                            Comment


                              #15
                              imported post

                              In your example, is the last data (Format Data Series) set to today? If yes, then this is correct behaviour. NT will always load the current days data into the chart on start of a new connection if the last day is set to today.

                              Ray
                              RayNinjaTrader Customer Service

                              Comment

                              Latest Posts

                              Collapse

                              Topics Statistics Last Post
                              Started by RB104, Today, 06:06 PM
                              1 response
                              10 views
                              0 likes
                              Last Post NinjaTrader_ShawnB  
                              Started by jtrading82, 05-25-2018, 09:59 AM
                              9 responses
                              978 views
                              0 likes
                              Last Post NinjaTrader_ShawnB  
                              Started by syxforex, Today, 12:11 AM
                              2 responses
                              15 views
                              0 likes
                              Last Post syxforex  
                              Started by aslane, Today, 02:31 PM
                              1 response
                              10 views
                              0 likes
                              Last Post NinjaTrader_Kate  
                              Started by stoner, 10-16-2019, 01:38 PM
                              5 responses
                              53 views
                              0 likes
                              Last Post sledge
                              by sledge
                               
                              Working...
                              X