Strategy Optimization - Crashing NT 8

[NTPlayer]

Is anyone else having problems with NT 8 crashing hard when running custom strategies in the Optimization? I have 3 custom strategies created for me by 3 different people and they all crash NT. I've purchased a new computer and refreshed everything from start twice... If a optimization is going to run for more than 5 minutes, there's a 30% to 50% chance that it'll freeze or crash my computer. It makes it unbearable to try and test backtest strategies.

Thoughts??

[cincai]

Freeze mean hang ???

Will it be because too lag ??
I mean too big data to load but your computer RAM is too low

[cincai]

Why dont you just test 1 strategy only ??

[honest-trader]

Yes i faced the same problem maybe the software NT8 is exhausted...but thats the reason it was purchased
My system is mint fresh condition as it is not even a week since it was built, 32 gb ram, i9 processor...

[NinjaTrader_Jim]

Hello NTPlayer and honest-trader,

A crash would mean the platform shuts itself down.

Messages stating NinjaTrader is not responding would be seen when memory is exhausted, and is waiting for garbage collection to decommit resources before new resources can be committed to continue the optimization.

I suggest opening your Windows Task Manager, viewing the Performance tab, and then right clicking on the graph to view all logical processors. When you reproduce symptoms, look here to see if a CPU core is maxed out, or if your memory is maxed out. If one core is maxed out, that would mean one specific optimization iteration is highly CPU intensive and the optimization would be waiting for that iteration to complete before continuing.

I have included some information for the thread's reference that explains memory utilization in the Strategy Analyzer below. Included are some tips for running efficient optimizations.

Understanding Optimization Memory Utilization

I have created a video demonstrating how the SampleMACrossover strategy, a simple strategy that utilizes IsInStantiatedOnEachOptimizationIteration=false for efficiency, can still quickly utilize memory resources.

Demo — https://drive.google.com/file/d/15pz...w?usp=drivesdk

We should consider the following for memory consumption:

Data * Strategy resources * Number of optimization iterations * Number of trades * Keep best # of results.

As we can see there are a number of factors that are involved and memory utilization can climb very quickly depending on a few of these factors. Once memory gets maxed out, we can experience short freezes where memory is decommitted, stored to disk, and then new resources are committed before the backtest is resumed.

We can easily control the number of iterations involved and we can also consider writing our strategies to use IsInstantiatedOnEachOptimizationIteration = false; (Which requires that we reset class level variables in State.DataLoaded.)

IsInstantiatedOnEachOptimizationIteration — https://ninjatrader.com/support/help...niteration.htm

Optimization Tips — https://ninjatrader.com/support/help...ionPerformance

Walk Forward Optimization — https://ninjatrader.com/support/help...ss_metrics.htm

Genetic Optimization — https://ninjatrader.com/support/help..._algorithm.htm

Please let us know if you have any additional questions.

[NTPlayer]

So I'm only running 1 strategy at a time and when it freezes (NT stops functioning, even strategies running stop and I can't watch open positions.. NT is frozen indefinitely), I look at the CPU usage and NT may be even under 10% with most memory freed up. So when it freezes and I check the memory usage, it's not overwhelmed at that point. Even if it was overwhelmed at some point, when it calms back down to under 10% (for instance), it should free back up - but it doesn't. It stays locked up until I do a hard exit on NT8. Something happens that causes it to stop completely. Once this happens once, it happens more frequently until I reboot my computer. This has happened on 2 computers and I've refreshed (started NT over from scratch) a couple times. It can't be a strategy issue because 3 different ones it happens too and it can't be a computer issue, because this has happened on 2 different systems. NT must not be handling more complex strategies well. It's frustrating while I try to find the best settings.

[NinjaTrader_Jim]

Hello NTPlayer,

As we can see in my demonstration video, even simple strategies like SampleMACrossover can quickly use up resources, this is especially the case if a lot of data is used and if many trades are calculated.

Optimizations are in nature resource intensive, they can and will use all of the CPU and memory resources available. It would be ill advised to run optimizations when trading live because you would need the CPU resources to ensure the live strategies trade properly.

I would suggest doing your optimizations and backtests when you are not trading live, and I also would suggest ensuring that all resource intensive applications are closed when you are using the Strategy Analyzer so you know it has all of the resources it can get. You may also perform the same test in the demonstration video I have. When memory gets fully utilized, it pauses and waits for garbage collection to free resources and allow new resources to be committed to memory and have the strategy continue.

You should have the Windows Task Manager open so you can view the CPU load on each processor and the full memory utilization. When the freezing symptoms occur, check the logical processor workload and the memory utilized. This would show where the bottle neck is at for you.

I also suggest looking into Genetic Optimizations and to follow the tips for efficient optimizations linked in my last post before considering to purchase new hardware if you are considering upgrading your PC so you can more easily perform these resource intensive tasks.