NinjaTrader crashes while loading historical data

[okui88]

thanksyou already shared

thanks for share news the forum may be better to share

[NinjaTrader_PatrickG]

I can confirm that the ever-increasing memory use when downloading historical data is also an issue for the 64-bit version of NT8 -- at least when it is running on Windows 7.

On each successive symbol or date range that is downloaded, the memory required by NT8 is taken from the system and added to the memory already reserved for the prior download. This cycle repeats until the system literally runs out of RAM. As a result, NT8 freezes up (as does any other app running on Windows) or becomes unstable. At that point, the only way out is to kill the program in Task Manager as the Control Center no longer responds to user input.
.

Can you provide me with exact steps to recreate the crash you're describing while using NinjaTrader 8 64-bit?

[D Trader]

Can you provide me with exact steps to recreate the crash you're describing while using NinjaTrader 8 64-bit?

1. Find a Windows 7 PC to run NT8. Issue has already been reported for other OS, however, so Windows version is probably not a key factor.
2. Select "Import Historical Data" from the Control Center menu.
3. Bring up Windows Task Manager, and note the memory used by the NinjaTrader.exe process before starting to download.
4. Begin downloading historical ES, TF, YM, etc. data. I typically download one month at a time.
5. Note the ever-increasing NT8 memory consumption in Windows Task Manager.
6. Immediately perform another download, same or different instrument.
7. Memory consumption continues to build.
8. Now take a break. Wait about 5 minutes, and NT8 will release the memory back to the OS.

So the issue is: Freeing up memory does not occur at conclusion of each download, but after some random period of time. Most users don't wait 4 or 5 minutes between downloads (not a reasonable thing to do).

Steps 4 to 6 can be repeated until all physical and virtual memory is tied up. Naturally, Windows will drop to its knees when this happens, and all applications will become unresponsive, giving the impression that NT8 has crashed. It has not (yet)... it just does not have the resources to respond to user input.

Does this help?

[NinjaTrader_PatrickG]

We have confirmed the crashing behavior on NinjaTrader 8 32-bit, so what you're reporting would not be the same issue as you've stated that you're using NInjaTrader 8 64-bit.

The memory being released is controlled by your operating system. when importing historical data, the data gets loaded into the memory as it is written to the disc and then memory will free up at whatever time your operating system releases it. If you see a large amount of RAM being used when importing, you will want to wait until the memory is released to continue.

Also, I was unable to recreate a NinjaTrader 8 64-bit crash following these steps:

• Clear out all cache and historical data information
• Open NinjaTrader > Tools > Historical Data > Load tab
• 'select all' Futures built-in instrument list
• Choose 'Tick' only
• Set the date range for 1 year ago
• Download

While this did take a large amount of my available CPU and RAM, it did not freeze or crash unexpectedly.

If NinjaTrader 8 64-bit is crashing in this scenario, I'd like to get more information from your NinjaTrader and your PC. Write in to PlatformSupport [AT] NinjaTrader [DOT] com with a link to this forum post and reference 1620093 ATTN Patrick G

[D Trader]

Patrick, thanks for looking into this.

However, your explanation really doesn't fly. Other applications and platforms import larger amounts of data than what is being requested of NT 8, and they have no problem at all with memory management.

The key phrase here is "data gets loaded into memory." Correct -- but not all of the data all at once! NT 8 should try chunking the data into more manageable segments, with a cap on total memory use. Furthermore, once the data is written to disc, it no longer needs to reside in memory. Therefore, the memory can be re-used.

Good grief -- I can download a GB of data in minutes using just a compiled C app and 64K of extra RAM! So I am 100% confident that NT can do even better.

So it's back to NT Support. A modification to code is clearly required. This is not a user issue, and there is no reason anyone should have to fuss with Task Manager, clear their cache, or restrict the types of data downloaded to avoid a Windows "out of memory" error!

Lastly, I didn't say anything about a "crash." Windows can hang for days when out of memory, but never crash. And frankly, I have never run any other application that consumed as much memory as quickly as NT 8 when downloading data.

Please let the community know when this issue is resolved. Thank you.

[sledge]

Good grief -- I can download a GB of data in minutes using just a compiled C app and 64K of extra RAM! So I am 100% confident that NT can do even better.
.

For my curiosity - how much ram do you have on this Win7 box? (I might never hit this condition with some of my Win7 boxes).

Have you disabled the page file?

[D Trader]

This particular PC has 8GB ram -- 6GB free for NT8 use.

[NinjaTrader_PatrickG]

Patrick, thanks for looking into this.

So it's back to NT Support. A modification to code is clearly required. This is not a user issue, and there is no reason anyone should have to fuss with Task Manager, clear their cache, or restrict the types of data downloaded to avoid a Windows "out of memory" error!

Please let the community know when this issue is resolved. Thank you.

I will submit a feature request for the Development Team to look further into the memory usage and management by the platform.

[traderpards]

Patrick, can you add the Strategy Analyzer (optimization) to your request? It too jacks memory usage beyond any comprehensible expectations. I keep waiting version after version for someone to actually fix that and no one ever does. Thanks.

[NinjaTrader_PatrickG]

Patrick, can you add the Strategy Analyzer (optimization) to your request? It too jacks memory usage beyond any comprehensible expectations. I keep waiting version after version for someone to actually fix that and no one ever does. Thanks.

Can you give me steps to reproduce the large memory consumption so I can test this, compare it with the expected behavior, and submit it if it is unexpected?

[tyrin_snow]

NT 8.0.2.0 connected to Kinetick Live Data Feed

Tried to update my historical FOREX charts via the Historical Data Manager. For updating I selected the instrument list "FOREX", which contains the following instruments

- AUDUSD
- EURCHF
- EURGBP
- EURJPY
- EURUSD
- GBPUSD
- USDCAD
- USDCHF
- USDJPY

System.OutOfMemoryException was unhandled
Message: chatroulette An unhandled exception of type 'System.OutOfMemoryException' occurred in NinjaTrader.Core.dll

Details in the trace file shown were:

omegle website Physical memory used was only 4.3 GByte from 8.3 Gbyte of total physical memory of my PC. However, I could observe that the RAM went up by 1GByte during the loading of historical data.

NinjaTrader was entirely frozen and I had to close via the Process Explorer.

Afterwards I got an error message: "Unable to recover NinjaTrader custom assembly. Please reinstall NinjaTrader."

All this happened just, because I started downloading historical data.

Did you also test the 64-bit version?

[NinjaTrader_Jason]

Hello tyrin_snow,

This forum thread handles version 8.0.2.0 and is also almost 5 years old. Can you please clarify why you post on these old forum threads? Are you running NinjaTrader 8.0.1.0 or 8.0.2.0?