• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Historical Data Manager memory consumption

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Historical Data Manager memory consumption

    i want to download a set of about 1,500 symbols that trade on the NYSE. i have been trying to accomplish this for 3 or 4 days now. i have my 'nysesymbols' symbols list imported via the symbol import utility. i go to the Download tab of Historical Data Manager, left click on my nysesymbols list, specify my start date of 1/1/2004 and end date of 3/21/2014, check Minute, Day and Last and click Download. I see status messages for the individual symbols on the bottom right of Control Center, so at this point it seems as though things are progressing just fine.

    the problem is that if i pull up Windows Task Manager and watch the Performance tab i see memory consumption increase linearly with time until after about 30 - 60 minutes all 8GB of system memory are consumed. at this point the app and machine become sluggish. i've tried a couple different things. i've tried doing 'abort', then once that finishes up i reboot the machine and then relaunch the same download job. problems with this approach are:

    1) risk of corrupting the data base if the abort doesn't go well
    2) symbols are not downloaded in alphabetical order nor does the app give any indication of progress of the overall download job so i have no way to know if i'm making my way through the download job or if i just keep redownloading the same symbols every time i restart but after 3-4 days of this i am concerned it's the latter.
    3) i am restricted to downloading during hours when i am awake and available to reboot my machine every 30 minutes. that is, i can't do downloads overnight while i sleep cause it pegs memory and seems to stall eventually. plus i'm concerned that it's thrashing on my solid state drive which could wear it out prematurely.
    3) it's inconvenient

    i have tried just letting it run overnight and that just stalls a some point in the night. i have to kill the entire NT process and reboot machine to get things going again.

    A) is this normal/expected?
    B) is there a way to know status of the overall download job?
    C) when i 'abort' then reinitiate the download on this same instrument list does it start all over from the beginning or is it smart enough to know that some of the symbols are already downloaded/saved and don't need to be redownloaded?

    setup: NT 7.0.1000.22 64 bit, Kinetick paid subscription

    #2
    Hello upshot,

    Thank you for your post.

    This is not expected behavior. You should see information on which instrument is downloading and what time period on the Control Center's bottom right hand corner next to the 'Abort' option. Aborting and restarting the process will have the process start from scratch.

    So I may investigate this matter further please advise how many Workspaces you have open (File > Workspaces) and the size of your database file which can be found by going to Documents-->NinjaTrader 7-->db folder and the database will be named 'NinjaTrader' and will be a .SDF file type.
    Patrick H.NinjaTrader Customer Service

    Comment


      #3
      Patrick,

      I do see information on which instrument is downloading and what time period on the Control Center's bottom right hand corner next to the 'Abort' option as you describe. It's just that it will give info about like AMZN, then maybe ITW, then BA, then ZION etc. In other words, it's not alphabetical so I can't tell, after say 5 hours, what percentage of the 1500 symbols have been downloaded.

      when i right click | properties on my \db folder i see:
      Size: 4.79GB
      Size on disk: 10.2 GB
      Contains: 2,835,246 Files, 2,586 Folders

      Comment


        #4
        Hello upshot,

        Thank you for your response.

        Please go to the following directory on your PC: (My) Documents\NinjaTrader 7\db --> then locate the file named 'NinjaTrader', it will be amongst the 'data', 'minute', 'tick' and 'day' folders. Right click on the file named 'NinjaTrader' then select Properties and advise the size of the file.

        In addition, please send me your log and trace files for today so that I may look into what occurred. You can do this by going to the Control Center-> Help-> Mail to Support. Please place 'ATTN: Patrick' in the subject line and reference the following ticket number in the body of the email: 1044646
        Patrick H.NinjaTrader Customer Service

        Comment


          #5
          oops, i botched that first response, didn't read carefully enough. i only have 1 workspace open and my ninjatrader.sdf file is 390MB. now i will email you the log and trace files. thanks!

          Comment


            #6
            Patrick, I've tried several times to email you the log and trace files as you said but I keep getting an error message that reads: "Failed to send mail. a connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 64.57.210.216:25"

            i rebooted my machine and tried a couple more times to same effect.

            Comment


              #7
              Originally posted by upshot View Post
              oops, i botched that first response, didn't read carefully enough. i only have 1 workspace open and my ninjatrader.sdf file is 390MB. now i will email you the log and trace files. thanks!
              Does it work if you request 1,500 symbols for 1 year or data instead of 10 years of data?

              Does 3 years work? 5?

              What about 7,500 symbols for 10 years?

              Does 100 symbols for 10 years work?

              Comment


                #8
                Hello upshot,

                Thank you for your response.

                Please manually attach your most current log and trace files to your e-mail to support[at]ninjatrader[dot]com, so I may investigate this matter further.
                • You will find the log file on your PC in the (My) Documents > NinjaTrader 7 > Log folder.
                • The log file will be named "log.20140327.txt"
                • You will find the trace file on your PC in the (My) Documents > NinjaTrader 7 > Trace folder.
                • The trace file will be named "trace.20140327.txt"
                • Please place 'ATTN: Patrick - 1044646' in the subject line.
                Patrick H.NinjaTrader Customer Service

                Comment

                Latest Posts

                Collapse

                Topics Statistics Last Post
                Started by Nate G, Yesterday, 04:21 PM
                0 responses
                18 views
                0 likes
                Last Post Nate G
                by Nate G
                 
                Started by Jschmelz, Yesterday, 04:45 PM
                1 response
                18 views
                0 likes
                Last Post NinjaTrader_Marco  
                Started by sisenberg, Today, 07:29 AM
                1 response
                9 views
                0 likes
                Last Post sisenberg  
                Started by Oceans, Today, 02:22 AM
                2 responses
                25 views
                0 likes
                Last Post Oceans
                by Oceans
                 
                Started by sisenberg, Yesterday, 08:06 PM
                1 response
                25 views
                0 likes
                Last Post sisenberg  
                Working...
                X