NT8 Optimizer uses up memory and then locks up

[SapientTrader]

So, I when I optimize a strategy in NT8, it works pretty well until it uses up about 90% of memory (20Gigs or so) and then it has a hard time finishing. I assume some garbage cleanup is happening but I have to limit the number of iterations I am running in order to complete a test.

I should also mention that prior to this, the optimization uses all of the CPU and then after memory fills up, the CPU usage goes way down. From time to time, the Memory usage will drop a bit which I assume is garbage collection. But the memory hovers at max and the iterations slow way down at this point and take a lot longer to complete. With now way to combine iteration results, it can be difficult to get a picture of how the strategy operates.

Workarounds include reducing timeframe covered, higher timeframes for the strategy itself, setting the parameter for https://ninjatrader.com/support/help...niteration.htm

I was having some errors when I set that to false before. I will try to clear my variables better.

I assume others have these same issues. Barring a major overhaul of how the optimizations work, are there other things I am missing?

My code is pretty clean that does not call functions unless needed. Thanks in advance y'all. I did some searches and didn't come up with much which surprised me and made me wonder if I didn't do the search right.

[SapientTrader]

Perhaps a better question is what factors within the strategy or indicators called by the strategy lead to memory bloat? Number of parameters being optimized for? Grain of data of the bars being tested?

[NinjaTrader_Jim]

Hello SapientTrader,

Thanks for opening the thread.

Your understanding of the garbage collection, clearing variables and using IsInstantiatedOnEachOptimizationIteration = false for optimization backtests is correct. For simplicity, let's call this IIOEOI.

Setting IIOEOI to false is the advised solution to writing a strategy so it can be optimized without causing drains in memory and performance. We can see that garbage collection does eventually kick in when this is set to true, but waiting for garbage collection to free resources is not particularly ideal when performing long optimization backtests.

The factors that contribute to extreme memory usage will be associated with the number of variables used in the strategy and the indicators it uses, compounded by the number of optimization iterations. Data series granularity and amount of data loaded will factor into memory usage, however, the way we manage variables will have the largest difference in performance that we can control.

The documentation page in your post can provide some examples so you may compare performance using IIOEOI = false/true and following the best practices. I'll also include a link to our documented best practices which touches more on marking variables to be cleared by garbage collection.

NinjaScript Best Practices (Performance) - https://ninjatrader.com/support/help...tm#Performance

Please let us know if you have any additional questions.