Memory Consumption exploding

[Beauregard]

Hello

When I test my strategy with the optimizer with many parameters, the memory consumption is just exploding. It grows per second ca. 3 MB. I disabled all my output to files or the information box. Is there any way to "clear" the memory when one step in the optimization has been executed?

thanks
Andre

[NinjaTrader_Brett]

Hello,

Are you outputing anything to the NinjaTrader log? Using Log()? Or what methods are you using?

I look forward to assisting you further.

[Beauregard]

no, nothing. I have a few print-statements, but the output window is not open. could this have an effect?

[NinjaTrader_Brett]

Hello,

Ok in this case there is nothing you can do to remove the data. If you where using log() this would require manual clearing this is why I asked if you are using it. However NinjaTrader uses C# and C# manages the memory NinjaTrader uses and has whats called a garbage collector that will release the memory when it is needed.

Unless your using some unsupported c# methods that we do not support that require manual Dispose().

Let me know if I can be of further assistance.

[Beauregard]

Hi
I investigated more. The problem is, that when you use arrays of struct in the strategy and run the optimizer, the memory is not disposed. I guess this happens with all variables.
In my OnTermination() procedure I added now this:
MyArray=null;
System.GC.Collect();

This helped. Memory is still growing, but much slower and from time to time it even shrinks

[Trader.Jon]

Beauregard,

Thanks for this tip! Could you please provide just a bit more details? I am guessing that

1. 'MyArray' includes all the price related DataSeries in "protected override void OnStartUp()"
2. that you added something in the "Using Declarations"
3. I have never used the "OnTermination" and there is little in the documentation... is there other related entries in that section that are helpful to this issue?

TIA!
Jon

[Beauregard]

Jon

1. With 'MyArray' I mean an array I used myself, such as:
private struct Capital
{
public DateTime Date;
public string CurrentPos;
public double CapitalClose;
public double CapitalMin;
public int TradeNo;
public double LastKapital;
};
private Capital[] CapitalArr = new Capital[10000];

I assume that the memory consumption, which is not disposed naturally, not only for array applies but also for all other variables.
c# has a built-in garbage collector. In principle it is not needed to call it manually, as I read in other forums. But as it seems, running the optimizer in NT, the garbage collector is not called when one run is finished, so it helps to call it manually. I cannot say whether it is needed to set the array to NULL before.
2. I have not additional using declarations related to this issue
3. In fact, I also hardly found any information about OnTermination(). I found it when I tried to find a way to execute some commands after the last trade. See here http://www.ninjatrader.com/support/f...ad.php?t=35853

Andre

[neb1998]

Brett, can you give an example of an unsupported method?

I am having similar issues with a new system i wrote....Memory is maxing out at about 100% of system usage and then the cpu cycles slowly drop to zero. I have tried turning off order history and it makes no difference so i am guessing i have a problem somewhere.

I have no print or log statements and only am looking at a year of 1 min bars for testing.

i have some custom indicators but they are running simple math, no loops, one dataseries total.

??

[NinjaTrader_Bertrand]

It would simply mean using a non documented method as per our officialp helpguide / NinjaScript language reference. Can you reproduce the issue? Did you try isolating out if one timeframe / barstype / market / indicator in particular would give rise to the issue?

[neb1998]

Yeah i am just using 1 min bars and 15 minute bars, i took out those entrances to just use 1 min bars and i am having the same issue. Each time i try to run the optimizer it spikes to max memory in a few minutes.

I cant imagine its the exits but i might try disabling them one by one.

[NinjaTrader_Bertrand]

Ok, please let us know your findings - or if there's a script you would like to us to test please either attach or submit to support - thanks.

[neb1998]

To add to this, the more cores you run ==more memory needed. If you have a machine with Hyperthreading and less than 6gb of ram you may want to turn it off. It will solve many of the out of memory problems.

Also decrease the population size and increase # of generations.

[neb1998]

There is def some kind of issue with optimizing, some strategies work fine on some tests and then the same code ran later will quickly eat up all system memory and pretty much stop computing.

[neb1998]

If my code is causing the memory problem, what is the best way to go about debugging code run through an optimizer or WF?

usually the problem happens in the first few seconds....traceorders or what?

Help!?

[NinjaTrader_Brett]

Hello,

We would simply need to comment out sections of your code until we find the cultprit for the large memory consumption.

You can choose to do this on your own by commenting out sections of the code and continually running until it no longer uses up memory.

Or you can send the strategy to me and I can do this for you.

Please post the strategy here or send it to support at ninjatrader dot com ATTN Brett and reference this forum post.

I look forward to assisting you further.