Importing a huge amount of historical data from a file (FYI)

[xTrader1]

Hello,

I tried to import 10 years of historical tick data for EUR/JPY . It was too much for Ninja , eventually the system entered thrashing ( for non-tech , a situation when pieces of memory are copied from RAM to disk and vice versa , the system cannot do anything useful and eventually freezes).

I have 8GB of physical RAM , with committed memory of16GB .
During importing , NinjaTrader took > 12GB of committed memory , working set was between 5-6GB.
I killed NinjaTrader process , restarted my computer and now I am importing this 10 years data piece by piece, which is feasible.

I guess , Ninja is trying to import all my data at once.
I don't complain , I understand I put Ninja under very heavy load.

However it would be good to publish recommendations regarding amount of data to be imported at once.

It would be even better to change mode of importing - once Ninja importing mechanism sees there is too much data to import at once - importing will be done piece by piece internally. Please transfer this suggestion to developers.

Thanks.

[NinjaTrader_Kate]

Hello xTrader1,

Thank you for your reply.

What you're seeing is occurring when physical memory is fully used and the page file (virtual memory) starts being used.

There is no limitation on how many lines can be in the text file you import, however you may notice that a large file takes some time to import. It would be best to ensure you are not connected to any data providers while importing the text file, this will free up NinjaTrader to process the import faster if speed is your concern. NinjaTrader 8 is multithreaded and will use as much of your PC resources as it can get when performing intensive activities like importing files that may be extremely large. It would also be recommended not to be running any other programs when trying to import very large files.

That being said, I have created a feature request on your behalf to improve the user experience when importing large data files. This request is being tracked under the number SFT-4933.

As with all feature requests, interest is tracked before implementation is considered, so we cannot offer an ETA or promise of fulfillment. If implemented, it will be noted in the Release Notes page of the Help Guide.

Release Notes — https://ninjatrader.com/support/help...ease_notes.htm

Please let us know if we may be of further assistance to you.

[xTrader1]

OK , so I'll share my understandings with the community. Typical tick data for FX pair has a few millions, sometimes even more more than 10 million ticks. For such a data Ninja requires 4-5 GB memory. This is feasible for computer equipped with 8GB RAM. So eventually I imported 10 years data of 3 FX pairs piece by piece ,doing import 30 times.

May be importing historical data onto fresh database requires less memory. As for now I don't recommend importing more than 1 year of historical data at one.

[afcmaff]

Just to add some 'hopefully useful' content to this thread on the subject of the data import experience and overall NT platform performance. I'm running a pretty powerful setup (AMD 3990X 64-Core with stable OC at 4.4GHz + 256GB RAM OC'd + fast-IO M.2 storage drives) with NT latest 8.0.25.0 build. I am currently importing a 36GB Tick Data LLC file (ES tick-by-tick last trade) as "ES ##-##.Last.txt" and as expected it takes some time to load (6+ hours and counting so far) and NT8 is using an average 85 GB of RAM for that job.

The main thing that I feel would significantly improve user experience is to display a progress bar with estimating time remaining so that we know nothing is frozen or stuck with the import -- for now you have to assume it's working by checking the Task Manager in Windows. So add my vote to SFT-4933 if that can be added into the feature request.

As far as overall performance, my only gripe is with NT8 multi-threading limitations... a real downer when you have 64-cores and only a few are being used by NT. The limitation in question is instrument-based. For example, I mainly trade ES, so I like to have various charts of the same instrument but having only a single instrument will not take advantage of multi-threading as much as multiple instruments. Perhaps it's more of an architecture issue (monolithic vs microservice), but I'm hoping NT is moving towards faster, reliable, and more robust multithreading.

[NinjaTrader_Kate]

Hello afcmaff,

Thank you for your reply.

I have added a vote and your comments to SFT-4933.

Please let us know if we may be of further assistance to you.