Memory Leak in Strategy Analyzer

[NinjaTrader_Jim]

Hello everyone,

There seems to be a bit of confusion for what is happening here.

Yes optimizations use a significant amount of memory and yes, the memory is not decommitted right away.

NinjaTrader 7 had a way of forcing garbage collection and NinjaTrader 8 just lets .NET handle it.

Peter, when you wrote in over email, you gave a us a video showing your test with the SampleMACrossover strategy where it used a large amount of tick data to build 200 volume bars. That will have a noticeable impact on memory which we do see in your video. I performed this test, and then I continued testing to see if the memory was in fact not getting decommitted. After completing your test, I downloaded the same amount of data for ES 06-20 and before I ran the same optimization backtest on that data, the memory was decommitted. This tells me that .NET garbage collection is working and there is not a memory leak.

Something else to consider with optimization backtests: By default NinjaTrader will create new instances of the strategy for each optimization iteration. A strategy can be programmed to use IsInstantiatedOnEachOptimizationIteration=false (class level variables will need to be reset in State.DataLoaded) and the optimizer will not create new instances of the strategy for each optimization iteration. The additional optimization iterations will increase the amount of memory utilized, and will increase the the time needed to complete the backtest.

AI Generate strategies do not generate code that is compliant with IsInstantiatedOnEachOptimizationIteration=false, so AI Generate runs will carry a large processing time in nature.

SampleMACrossover utilizes IsInstantiatedOnEachOptimizationIteration=false, but I thought this would be worth mentioning for others running optimization backtests.

IsInstantiatedOnEachOptimizationIteration - https://ninjatrader.com/support/help...niteration.htm

Peter, if there is a test similar to the video provided where we can observe that memory is not decommitting after a period of time, we can look into that.

Otherwise, what you would be looking for is to have NinjaTrader implement a way to force garbage collection. We do have a feature request tracking interest behind this with the ticket ID SFT-2111. Let me know if you would like us to add a vote on your behalf.

I look forward to assisting.

[peterslane1]

Ok, yes please add my vote for the ticket ID SFT-2111 to have NinjaTrader implement a way to force garbage collection.

Thank you for looking into this and explaining it.

Anyone else who wants this feature please add your vote to this thread.

[peterslane1]

I am getting resource exhaustion and ninja hang while doing optimization in Strategy Analyzer still using volume bars.

This issue has not been resolved!

I have upgraded to 128 GB DDR4 RAM.

Ninja still managed to use all of the memory plus exhausted the virtual memory.

I get a resource exhaustion error. There is also an error with NET UltraView.

I have included the error message of NET Framework Ultraview.
There are others of ninja application hang and resource exhaustion error which I will attach in next support emails.

We have already gone through all of your safe mode steps before and support has verified this is an issue in the previous post. Your response previously was for me to vote for a feature request to force garbage collection, which I did. I think that is a ridiculous response to an issue that you acknowledge.

How will voting for a feature fix the functional problem? Seriously.

[peterslane1]

I understand this is a garbage collection issue. Here are some specific inputs that seem to affect the resource exhaustion due to limitations of the garbage collection within NET Framework:

1. Using volume bars
2. Multiple runs. Exhaustion may occur on the 3rd or 4th run. Do I need to close the strategy analyzer after each run or close the workspace as one user suggests after each run?
3. Optimizing data series

For other user's sake how does one effectively work around the limitations of the NET Framework's garbage collection limitations?

[NinjaTrader_Jim]

Hello peterslane1,

You wrote in over email and brought up an issue where resources are not freed when using Optimize On Data Series and using Volume bars. Our Quality Assurance team is investigating that further.

We will continue to work with you over email and will post any necessary updates here.

[Jzgmma]

Ok, yes please add my vote for the ticket ID SFT-2111 to have NinjaTrader implement a way to force garbage collection.

Thank you for looking into this and explaining it.

Anyone else who wants this feature please add your vote to this thread.

Its now 18 months later and still not working! Anything with tick data or even minute data with lots of data point to analyse on Strategy Analyzer is very very frustrating!! Almost useless, will this ever get fixed ?

For what its worth, YES PLEASE add my vote to ticket ID SFT-2111 !!!

[NinjaTrader_Jim]

Hello Jzgmma,

We have added a vote on your behalf for improved garbage collection.

However, the issue is not so much a NinjaTrader issue so much as optimizations are extremely resource intensive.

Optimize On Data Series can also be extremely resource intensive in addition to regular optimizations.

I'm including a video that shows how resources may be quickly consumed even with a simple IsInstantiatedOnEachOptimization=false strategy like the SampleMACrossover strategy. This can help illustrate where memory consumptions goes in an optimization. Optimizations that create more trades will have higher memory consumption, and it will be best to configure optimizations that are manageable.

Understanding Memory utilization in the Strategy Analyzer

I have created a video demonstrating how the SampleMACrossover strategy, a simple strategy that utilizes IsInStantiatedOnEachOptimizationIteration=false for efficiency, can still quickly utilize memory resources.

Demo — https://drive.google.com/file/d/15pz...w?usp=drivesdk

We should consider the following for memory consumption:

Data * Strategy resources * Number of optimization iterations * Number of trades * Keep best # of results.

As we can see there are a number of factors that are involved and memory utilization can climb very quickly depending on a few of these factors. Once memory gets maxed out, we can experience short freezes where memory is decommitted, stored to disk, and then new resources are committed before the backtest is resumed.

We can easily control the number of iterations involved and we can also consider writing our strategies to use IsInstantiatedOnEachOptimizationIteration = false; (Which requires that we reset class level variables in State.DataLoaded.)

IsInstantiatedOnEachOptimizationIteration — https://ninjatrader.com/support/help...niteration.htm

Optimization Tips — https://ninjatrader.com/support/help...ionPerformance

Walk Forward Optimization — https://ninjatrader.com/support/help...ss_metrics.htm

Genetic Optimization — https://ninjatrader.com/support/help..._algorithm.htm

[DieSlower]

Was this ever addressed or fixed? I'm on NT 8.0.26 and just ran the SampleMACrossOver and still see my ram fly through the roof.
My own custom strategy does the same. We really need a way to make sure the RAM is cleared out in a timely manner.

[NinjaTrader_Jim]

Hello DieSlower,

The Strategy Analyzer keeps trades in memory. On systems with enough memory this makes optimizations very fast.

On systems with less memory, the number of trades, what is used by the strategy, and the data involved, can reach the PC's memory limits where we then see stalls and stutters.

What you can do in the near term:

1. Reduce Keep Best # of Results
3. If using Optimize Data Series, consider disabling to run smaller faster tests.
2. Program the strategy to use IsInstantiedOnEachOptimizationIteration = false (involves resetting class level variables in State.DataLoaded)
3. If Trade information is not needed, consider setting IncludeTradeHistoryInBacktest to false.
4. Consider using Genetic Optimizations to test less iterations.

In regards to SFT-2111 (improved garbage collection/better memory management in the Strategy Analyzer) I have added your vote.

Otherwise, an immediate solution would be to test on a system with more memory.

Recently, we noted some optimizations taking up to 24 GB of memory with the above factors set abusively. If you are able to test on machines with different amounts of memory available, you may find a sweet spot for having enough for your needs without going overboard.

[DieSlower]

Hello,

My system has 128GB of RAM, so it should have enough for this. I use about a week of data for my tests, but I do have a large number of parameters.
I keep 10 best results, so that should not be a problem, my iterations number in the millions though, so I need a way to let unused memory go.

I cant reliably use IsInstantiedOnEachOptimizationIteration because of the complexity of the simulation. Otherwise I would gladly use that option, although even with that on, I used to run into memory problems.

Thank you for adding my vote to the ticket. In the meantime, do you ahve some code example of how we can manually reduce memory after every strategy termination?

[NinjaTrader_Jim]

Hello DieSlower,

I do not have any other recommendations or sample code to provide to force garbage collection or to change the way memory is handled in NinjaTrader.

The juice may be worth the squeeze to make the strategy IsInstantiedOnEachOptimizationIteration = false compliant. You could test with it set to false, and you should see faster results (albeit incorrect if class level variables are not reset in State.DataLoaded.)

[gaz0001]

Any progress on this memory issue?

Running the first 2 or 3 Strategy Analysis goes quite quickly.

After a few more iterations, my laptop basically crumbles and becomes unusable

The performance gets bad after the memory gets to about 70%.

After some more time, stuff goes into the pagefile. When this happens, the machine is completely crippled.

The only way i found to get out of it, was to restart Ninja.

Its a pain.

[gaz0001]

Any progress on this memory issue?

Running the first 2 or 3 Strategy Analysis goes quite quickly.

After a few more iterations, my laptop basically crumbles and becomes unusable

The performance gets bad after the memory gets to about 70%.

After some more time, stuff goes into the pagefile. When this happens, the machine is completely crippled.

The only way i found to get out of it, was to restart Ninja.

Its a pain.

Now, the utilisation shows 88% used for the RAM.
However, the optimiser is completely useless. ITs not doing anything at all.

Trying to bring my Chart up, just has a blue spinney dot.
The software is completely crashed now.
The only way i can get out of this, is to Kill it in task manager.

[NinjaTrader_ChrisL]

Hi gaz0001, thanks for posting. The suggestions for improving performance have been given in post Strategy Development of this thread. As of right now, the feature request SFT-2111 is still in the processing state and I will add a vote to it for you.

Kind regards,
-ChrisL

[gaz0001]

Hi gaz0001, thanks for posting. The suggestions for improving performance have been given in post Strategy Development of this thread. As of right now, the feature request SFT-2111 is still in the processing state and I will add a vote to it for you.

Kind regards,
-ChrisL

Hi Chris,

I was looking for a workaround on the forum, and i can see people with the same issue dating back to May 2015.
This thread itself is March 2020.

Instead of raising it as a feature request, can it be raised as a bug? I believe it is a bug, and not a new feature.
I noticed today that even after you close everything down, the workspaces, strategy analyser etc, the RAM still doesn't clear. It stays on 80%. The whole NT8 platform then continues to run at a degraded performance.

The only way to get it back to normal speed is to shutdown the NT8 platform and reset it. This is a complete disaster for those running automated strategies.