Out of system memory when importing tick data

[filson]

Dear NinjaTrader,

I've begun testing NT7 with historical tick data.
For the EURUSD cross I've got 10 years or about 24.8GB of data.

When I tried to import this, NT7 ate all my system memory, to a full 7.4GB of used RAM.
Then stalled completely. Regrettably that was about 6 months before end of data.

Do you have any tricks for optimizing memory usage in NT7?
It seems like all data is kept in memory before flushing the final product as historical data.


Filip

[NinjaTrader_Matthew]

Hello,

With this amount of data, I'd recommend separating the data into smaller files and re-importing them one at a time.

I can't recommend a specific file size as the amount you can import at one time will depend on your system resources.

[filson]

Don't really think I can find an editor that will accommodate the file size.
I've already ordered more memory, hoping 32GB will fit it. Any ideas about chances on that?

Could be good to take a different approach to importing data, should you want to revisit the import code.

[NinjaTrader_Matthew]

Can't say for sure and I'm not sure I've ever seen this much data imported at once.

We are looking on improving the import methods in the next major release, but I do not have ETA on this at this time.

[filson]

Fair enough.

I much appreciate you taking this into account for future releases.
If you would like the sample data file for testing purposes, its about 524MB zipped.
I'd be happy to upload it somewhere accessible to you.

Filip

[sledge]

Don't really think I can find an editor that will accommodate the file size.
I've already ordered more memory, hoping 32GB will fit it. Any ideas about chances on that?

Could be good to take a different approach to importing data, should you want to revisit the import code.

Try textpad at textpad.com

You might need to setup a ram drive and do all sorts of stuff to load into memory.

Might be able to write a java/c#/c++ and truncate it. or use some unix command line trickery.

[sledge]

Fair enough.

I much appreciate you taking this into account for future releases.
If you would like the sample data file for testing purposes, its about 524MB zipped.
I'd be happy to upload it somewhere accessible to you.

Filip

There appears to be some products out there.

Good luck!

http://www.google.com/search?q=split+file

[filson]

Most editors i've come by, loads the entire file into memory rather than scan to current area. Maybe UltraEdit can do it. Notepad++ easily dies when loading large files, and its usually on par with TextPad.

I should get the extra memory tomorrow, so hopefully the import will be sorted with that alone.


@Matthew

Does NT7 support loading and doing historical testing on a data feed from a SQL database? It would truly be awesome if it could iterate through a simple select statement as data source.