Walk Forward Optimization & Memory

[philippe07024]

Hi there!
I use Win 7 64 with 12 GB RAM.
When I do a Walk Forward Optimization of a strategy over 6 months, using 13 Range bars and Optimization period of 28 days vs. Test period of 7 days, NT 64 uses all the memory available (~11.5GB) and then the WFO stalls, it runs for hours and hours with no error message.

Optimization period of 21 days vs. Test period of 7 days went through.

Does anyone have either a fix / work around or rule of thumb do define the acceptable ranges?

Philippe

[NinjaTrader_AdamP]

philippe,

There are many reasons this may be occurring, as it depends on how many variables you are optimizing over.

For example, if you have 6 variables being optimized on 20 steps each over a period of 31 days, you are effectively doing 6*6*20 runs of a strategy over 31 days. Since strategies can run on multiple threads when being optimized, you are probably maxing out your RAM due to having too many strategies running over a sufficiently long enough period of time. There unfortunately is no technical rule of thumb without delving into your strategies computational complexity, however you have basically done what I would have suggested, i.e. reduce time frame and/or reduce the range the variables are optimizing over and see if it improves performance.

Please let me know if I can assist further.

[philippe07024]

Correct me if I am wrong, 6 variables with 20 steps gives:
20x20x20x20x20x20=64,000,000 runs with default optimizer.

The strategy generates around 20 trades per months. I run an optimization on the Profit Target and Stop Loss, both 50 to 200 ticks in steps of 10. 15 x 15 = 225 combinations using the Genetic optimizer which should decrease the number of runs.
Rounding up, there are 20x225=4500 trades per period tested (in my scenario 21days for optimization + 7 days walk forward), over 6 months or 26 weeks = 117,000 trades. Nothing that should use 11.5 GB of ram, shouldn't it?

I reduced the "keep best # results" to 5 and the time frame as you suggested. Now the optimization goes through. However, at the end of the optimization the RAM is not freed up. So between 2 tests, I shut down NT and start it fresh.
Isn't there a possibility the the memory is not freed up as expected?

[NinjaTrader_AdamP]

Philippe,

There is more to consider. Let us fix 1 variable with a total of 6 variables being optimized and 20 steps for each. This 1 variable requires 6*20 steps to complete 1 step, but there are 6 of them total. In this instance, you have 6 variables fixed with 6*6*20 steps total.

[chrisrgr]

RAM usage is never freed

I have ninja 64 up-to-date. When I optimize a strategy, once it is done ninja never releases the RAM that it was using. So after optimizing a strategy so many times, ninja will become really slow and the Windows Task Manager will show that all RAM is maxed out.

I can only recover my RAM if I quit Ninja. Then RAM returns to normal usage levels and the process repeats itself the next time I run optimization.

I am running Windows 8 64bit with 6 GB of RAM. I only use this computer for Ninja.

Any suggestions?

[NinjaTrader_AdamP]

chris,

I am not seeing similar behavior here. I have been optimizing some strategies watching the RAM usage and see that it free's it up almost immediately after the optimization is finished.

Are you printing to the log or output window in your strategy you are backtesting?

[chrisrgr]

chris,

I am not seeing similar behavior here. I have been optimizing some strategies watching the RAM usage and see that it free's it up almost immediately after the optimization is finished.

Are you printing to the log or output window in your strategy you are backtesting?

Not that I know of. I am just testing strategies. Once I am done I have to quit ninja in order to free up my RAM. How can I check if I am printing a log file?

[NinjaTrader_AdamP]

chris,

You can check the Control Center > Log tab.

Also, how many strategy analyzers are you running at once?

[chrisrgr]

I am only doing one strategy. I can see single line entries on the log tab. I don't see if I am writing all my optimizing data results there.

[chrisrgr]

That would be awesome if it would free up. What other settings should I check?

[NinjaTrader_AdamP]

chris,

Are there like over 100 things in your log? Generally more would be required but I am mainly asking if there is a lot. This would be after you ran many optimizations, so it would require you to do what you did before to max out the RAM, then check the log.

After this, our next recourse is for you to send me your strategy along with a description of how to replicate your issue. If I still cannot, I would suggest a reinstall of Ninjatrader, i.e. uninstall it first then reinstall.

[AnotherTrader]

Which instrument are you testing on?

Tick compression might help ...

Search for "compress tick data for quicker backtests" in google ...

[chrisrgr]

I am testing CL. I have about 40 different settings in the strategy. The log file looks normal not too much stuff in it.

When you built a strategy are you suppose to release the RAM at the end of the code?

Once I am done with the strategy - I can quit the strategy analyzer and the RAM still stays maxed out. It is only when i quit ninja does it finally free the RAM. I'll look into compressed tick data.

[NinjaTrader_RyanM1]

Hi Chris,

Normally you do not need to do anything special but if your strategy uses custom resources, you need to dispose of them in OnTermination().

Examples of this could be seen in our samples that work with timer objects or streamwriter/reader.

http://www.ninjatrader.com/support/forum/showthread.php?t=3476

http://www.ninjatrader.com/support/forum/showthread.php?t=5965

[chrisrgr]

I have included the OnTermination() command in the Ninja Strategy and now I am able to quit during optimization where before I was not. However, the RAM is still full and stays that way unless I quit Ninja completely. I noticed there was a Dispose() command. Would this help my issue?