Historical Data Manager memory consumption

[upshot]

i want to download a set of about 1,500 symbols that trade on the NYSE. i have been trying to accomplish this for 3 or 4 days now. i have my 'nysesymbols' symbols list imported via the symbol import utility. i go to the Download tab of Historical Data Manager, left click on my nysesymbols list, specify my start date of 1/1/2004 and end date of 3/21/2014, check Minute, Day and Last and click Download. I see status messages for the individual symbols on the bottom right of Control Center, so at this point it seems as though things are progressing just fine.

the problem is that if i pull up Windows Task Manager and watch the Performance tab i see memory consumption increase linearly with time until after about 30 - 60 minutes all 8GB of system memory are consumed. at this point the app and machine become sluggish. i've tried a couple different things. i've tried doing 'abort', then once that finishes up i reboot the machine and then relaunch the same download job. problems with this approach are:

1) risk of corrupting the data base if the abort doesn't go well
2) symbols are not downloaded in alphabetical order nor does the app give any indication of progress of the overall download job so i have no way to know if i'm making my way through the download job or if i just keep redownloading the same symbols every time i restart but after 3-4 days of this i am concerned it's the latter.
3) i am restricted to downloading during hours when i am awake and available to reboot my machine every 30 minutes. that is, i can't do downloads overnight while i sleep cause it pegs memory and seems to stall eventually. plus i'm concerned that it's thrashing on my solid state drive which could wear it out prematurely.
3) it's inconvenient

i have tried just letting it run overnight and that just stalls a some point in the night. i have to kill the entire NT process and reboot machine to get things going again.

A) is this normal/expected?
B) is there a way to know status of the overall download job?
C) when i 'abort' then reinitiate the download on this same instrument list does it start all over from the beginning or is it smart enough to know that some of the symbols are already downloaded/saved and don't need to be redownloaded?

setup: NT 7.0.1000.22 64 bit, Kinetick paid subscription

[NinjaTrader_PatrickH]

Hello upshot,

Thank you for your post.

This is not expected behavior. You should see information on which instrument is downloading and what time period on the Control Center's bottom right hand corner next to the 'Abort' option. Aborting and restarting the process will have the process start from scratch.

So I may investigate this matter further please advise how many Workspaces you have open (File > Workspaces) and the size of your database file which can be found by going to Documents-->NinjaTrader 7-->db folder and the database will be named 'NinjaTrader' and will be a .SDF file type.

[upshot]

Patrick,

I do see information on which instrument is downloading and what time period on the Control Center's bottom right hand corner next to the 'Abort' option as you describe. It's just that it will give info about like AMZN, then maybe ITW, then BA, then ZION etc. In other words, it's not alphabetical so I can't tell, after say 5 hours, what percentage of the 1500 symbols have been downloaded.

when i right click | properties on my \db folder i see:
Size: 4.79GB
Size on disk: 10.2 GB
Contains: 2,835,246 Files, 2,586 Folders

[NinjaTrader_PatrickH]

Hello upshot,

Thank you for your response.

Please go to the following directory on your PC: (My) Documents\NinjaTrader 7\db --> then locate the file named 'NinjaTrader', it will be amongst the 'data', 'minute', 'tick' and 'day' folders. Right click on the file named 'NinjaTrader' then select Properties and advise the size of the file.

In addition, please send me your log and trace files for today so that I may look into what occurred. You can do this by going to the Control Center-> Help-> Mail to Support. Please place 'ATTN: Patrick' in the subject line and reference the following ticket number in the body of the email: 1044646

[upshot]

oops, i botched that first response, didn't read carefully enough. i only have 1 workspace open and my ninjatrader.sdf file is 390MB. now i will email you the log and trace files. thanks!

[upshot]

Patrick, I've tried several times to email you the log and trace files as you said but I keep getting an error message that reads: "Failed to send mail. a connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 64.57.210.216:25"

i rebooted my machine and tried a couple more times to same effect.

[sledge]

oops, i botched that first response, didn't read carefully enough. i only have 1 workspace open and my ninjatrader.sdf file is 390MB. now i will email you the log and trace files. thanks!

Does it work if you request 1,500 symbols for 1 year or data instead of 10 years of data?

Does 3 years work? 5?

What about 7,500 symbols for 10 years?

Does 100 symbols for 10 years work?

[NinjaTrader_PatrickH]

Hello upshot,

Thank you for your response.

Please manually attach your most current log and trace files to your e-mail to support[at]ninjatrader[dot]com, so I may investigate this matter further.

• You will find the log file on your PC in the (My) Documents > NinjaTrader 7 > Log folder.
• The log file will be named "log.20140327.txt"
• You will find the trace file on your PC in the (My) Documents > NinjaTrader 7 > Trace folder.
• The trace file will be named "trace.20140327.txt"
• Please place 'ATTN: Patrick - 1044646' in the subject line.