RAM Cleaner after optimization

[forallatlantis]

Hi there

I make some optimization and walk forward optimization but after one, max 2 checks I have to close NT8 and restart it in order to clean the memory. Even if I use soft that clean RAM it can't be done until I restart NT8. Is there any solution for auto-cleaning the RAM after an optimization or WFO is done?

Is there any way to make some optimization and make NT8 use less memory usage? I have 16 GB of RAM but it looks it's not sufficient.

Best regards
Claudiu

[NinjaTrader_JessicaP]

Thank you for your question forallatlantis.

We are constantly working to improve the efficiency and memory usage of NinjaTrader generally. From a user's point of view, the control you currently have available is to simplify your optimization runs. Since every combination of N variables optimized at the same time must be checked, optimization is by definition an O(2^N) operation. Fewer variables, and less information in your code sensitive to changes in variables (or sensitive variables tested by themselves), will enable you to optimize more quickly, and basket tests over a wide variety of instruments will ensure your strategy is robust and not overly sensitive to any particular set of market conditions.

Could I ask what you mean by having to clean the memory? If I can better understand this, I may be able to make a better recommendation to the product management team.

[forallatlantis]

Thank you for your question forallatlantis.

We are constantly working to improve the efficiency and memory usage of NinjaTrader generally. From a user's point of view, the control you currently have available is to simplify your optimization runs. Since every combination of N variables optimized at the same time must be checked, optimization is by definition an O(2^N) operation. Fewer variables, and less information in your code sensitive to changes in variables (or sensitive variables tested by themselves), will enable you to optimize more quickly, and basket tests over a wide variety of instruments will ensure your strategy is robust and not overly sensitive to any particular set of market conditions.

Could I ask what you mean by having to clean the memory? If I can better understand this, I may be able to make a better recommendation to the product management team.

I can successfully make optimization for a list of instruments but the entire issue is what after 2-3 optimization, depending on instruments, the memory still remain full and it cleans (free the memory) only when I close NT8. My logic says it would be cool if NT8 can free the memory it use after each optimization that is made.

[NinjaTrader_JessicaP]

Thank you for this additional information. Let's try to determine what is happening. Do you have any memory intensive programs you can open alongside NinjaTrader? If so, can you open and close one and check your memory usage after? If it drops down, this has to do with the way Microsoft .NET handles garbage collection. While we have some points of control, and will continue using the ones we have to the best of our ability, we must trust Microsoft libraries to handle many memory operations.

If you open a memory expensive application side-by-side with Ninja, close Ninja, and see no reduction in memory usage, or if there are any other questions we may answer, please let us know.

[forallatlantis]

Thank you for this additional information. Let's try to determine what is happening. Do you have any memory intensive programs you can open alongside NinjaTrader? If so, can you open and close one and check your memory usage after? If it drops down, this has to do with the way Microsoft .NET handles garbage collection. While we have some points of control, and will continue using the ones we have to the best of our ability, we must trust Microsoft libraries to handle many memory operations.

If you open a memory expensive application side-by-side with Ninja, close Ninja, and see no reduction in memory usage, or if there are any other questions we may answer, please let us know.

Hi there Jessica
I tried to encode a movie witch highly use CPU and RAM but after closing the app the system came to normal usage. I'd like to mention that after I close NT8 the system came to normal and free the RAM, but I'd like to ask if it's normal not to clean the RAM after a walk forward optimization is finished. After 1-2 WFO I have to close and reopen the NT8.
Best regards

[NinjaTrader_JessicaP]

I appreciate your patience with these questions. I realize it may seem like I am saying the same thing multiple times, but I need to be careful in my understanding before I can give advice.

Are you saying the only way to reduce memory usage after 2-3 optimizations is to close and reopen Ninja? Memory cleaner programs, and large programs next to Ninja, do not reduce memory usage? Or are you saying that memory cleaners and large programs do reduce memory usage, and that closing and reopening Ninja also reduces usage, and that this is what you mean by normal?

[forallatlantis]

I appreciate your patience with these questions. I realize it may seem like I am saying the same thing multiple times, but I need to be careful in my understanding before I can give advice.

Are you saying the only way to reduce memory usage after 2-3 optimizations is to close and reopen Ninja? Memory cleaner programs, and large programs next to Ninja, do not reduce memory usage? Or are you saying that memory cleaners and large programs do reduce memory usage, and that closing and reopening Ninja also reduces usage, and that this is what you mean by normal?

Hi there
The only way I can clean RAM is to close and reopen NT8. Memory cleaner programs like IOBit suite cannot do anything to clean RAM.
Best regards

[NinjaTrader_JessicaP]

Thank you forallatlantis. As a final question, are you able to cause this to happen with the SampleMACrossover strategy? If you are not, can you send us a stripped down NinjaScript code sample that we can use to make this happen on our systems?

[forallatlantis]

Thank you forallatlantis. As a final question, are you able to cause this to happen with the SampleMACrossover strategy? If you are not, can you send us a stripped down NinjaScript code sample that we can use to make this happen on our systems?

Same issue with SampleMACrossover. After the WFO has finished I wait for half an hour and RAM was still at 70% full but it clean immediately after I close NT8.
I have a print screen but I can't upload it here but on a URL.
Best regards

[NinjaTrader_JessicaP]

It looks like your memory cleaner won't be able to force garbage collection. When you analyze memory, please analyze only NinjaTrader's memory. You can do so with the Windows Task Manager (ctrl + alt + esc) . On our benchmarking systems the SampleMACrossover optimizing 1;10;1 and 1;25;1 on the S&P 500 as a basket test could not be pushed above 500 mb usage.

This publicly available link has more "under the hood" information.

https://stackoverflow.com/questions/1852929/can-i-force-memory-cleanup-in-c

[forallatlantis]

It looks like your memory cleaner won't be able to force garbage collection. When you analyze memory, please analyze only NinjaTrader's memory. You can do so with the Windows Task Manager (ctrl + alt + esc) . On our benchmarking systems the SampleMACrossover optimizing 1;10;1 and 1;25;1 on the S&P 500 as a basket test could not be pushed above 500 mb usage.

This publicly available link has more "under the hood" information.

https://stackoverflow.com/questions/...y-cleanup-in-c

I retest SampleMACrossover and I use historical fill processing to tick = 1 for higher precision. The memory usage jumped to 2.6 GB of RAM.The problem is not how high it jump but that it's not cleaning after finishing the optimization.
The force cleaning looks like I need to change the strategy in code right?
What's happening if not even SampleMACrossover is not working "properly" and not cleaning the RAM.
One last mention is that I made this test on 2 PC not only mine and the results are the same, so it could be a bug in NT8? Shouldn't NT force cleaning the RAM after an optimization is finished?
Best regards

[NinjaTrader_JessicaP]

To clarify the publicly available link, C# is a managed language. It is not possible to force .NET to clean memory, and the cleaning as soon as optimization is finished you are describing is not currently a feature Microsoft will allow through .NET . The method discussed simply queues the garbage collector take action the next cycle. Even if it does so, there are many memory references held by any program on your computer and the garbage collector must walk through these a step at a time, and may not optimize all memory in a single cycle.

If opening up several very large programs which push your memory usage near 16GB does not reduce Ninja's memory footprint after optimization, we would like to know as this has not occurred in our benchmark systems, and would point to too many handles being kept.

[rocker84]

Hi there

I make some optimization and walk forward optimization but after one, max 2 checks I have to close NT8 and restart it in order to clean the memory. Even if I use soft that clean RAM it can't be done until I restart NT8. Is there any solution for auto-cleaning the RAM after an optimization or WFO is done?

Is there any way to make some optimization and make NT8 use less memory usage? I have 16 GB of RAM but it looks it's not sufficient.

Best regards
Claudiu

Same problem here, and in this: https://ninjatrader.com/support/foru...-up#post827682

I run a backtest on the sp500 tickers, 6/10 times and then I got the ram increasing to 100% (30 GB). Once this happens the only way to free the RAM it is to close NT

[NinjaTrader_BrandonH]

Hello rocker84,

Thank you for your post.

My colleague Jim created a video demonstrating how the SampleMACrossover strategy, a simple strategy that utilizes IsInStantiatedOnEachOptimizationIteration=false for efficiency, can still quickly utilize memory resources.

Demo — https://drive.google.com/file/d/15pz...w?usp=drivesdk

We should consider the following for memory consumption:
Data * Strategy resources * Number of optimization iterations * Number of trades * Keep best # of results.

As we can see there are a number of factors that are involved and memory utilization can climb very quickly depending on a few of these factors. Once memory gets maxed out, we can experience short freezes where memory is decommitted, stored to disk, and then new resources are committed before the backtest is resumed.

We can easily control the number of iterations involved and we can also consider writing our strategies to use IsInstantiatedOnEachOptimizationIteration = false; (Which requires that we reset class level variables in State.DataLoaded.)

IsInstantiatedOnEachOptimizationIteration — https://ninjatrader.com/support/help...niteration.htm
Optimization Tips — https://ninjatrader.com/support/help...ionPerformance
Walk Forward Optimization — https://ninjatrader.com/support/help...ss_metrics.htm
Genetic Optimization — https://ninjatrader.com/support/help..._algorithm.htm

Please let us know if we may assist further.