Frying memory

[Frasson]

Hi Support Team
Summary of my problem:
I developed an indicator that works as a strategy (let's call ELTTS). It has 17 inputs that can be varied (setups) and two outputs. One output is a signal -1 or 0 or 1 that is to use for automation the other output is its own backtest (the strategy profit in the data range defined in the graph). So far so good, works great.
However as are many input set-ups it takes a lot of time to adjust the best set-up manually. To solve this problem I developed another indicator that tracks the best setups "to ELTTS" using "genetic algorithms". Everything is good so far.
During development I found that the NinjaTrader gets lost because of the "large memory consumption". Being that I was careful to always clean arrays, lists and arrays in addition to always break the references of the variables in memory (always insert (int) or (double) in front).
I verified that I can not leave the algorithm naturally converge because it generates innumerable loops and the NT can not take (consume all memory and locks).
Because of this I put a timer in the system, eg I set to 100,000 backtests and when it reaches "10 min processing" I force stopping the process. The resulting backtests the number depends on the size of the database. In this example got make 1000 backtests in 10 minutes for a small size base. When the processing time exceeds 10 min "the system stops" and gives me the results of the best tracked set-ups.
I identified that if I run the genetic algorithm immediately after starting the NinjaTrader the system promptly gives me the results in 10 minutes. However if after run once time the algorithm and after I rerun the algorithm genetic a NinjaTrader frie my memory and takes a lot longer than 10min to give me the answer and sometimes not respond.
I wonder what's going on? And if you have any tips to resolve this the problem of NinjaTrader memory fryer. kkkk
Another question
When i put values in the DataSeries that way myDataSerie.Set(value) is diferent than myDataSerie[0] = value?
Thank you

[NinjaTrader_Jesse]

Hello,

Thank you for the question.

Based on the description I would be unsure what may be occurring as I do not know what logic you have used in the files being run.

The only suggestion I could make based on the provided description would be to try and isolate the code or logic causing this. You could do that by reducing the code until the problem is resolved to find what area of code is potentially leading to this outcome. You would likely need to debug the code to find the answer to this.

If you are not able to re create the scenario using one of the sample scripts, it is likely something in the logic used in the script so you can use this as a test as well. If you are able to reduce and locate the logic causing the problem, I could look further into a sample of that logic to see what is happening.

Regarding the .Set, that is an internal method so there are internal processes that occur with that method opposed to using the other version.

I look forward to being of further assistance.

[Frasson]

Hi Jesse thanks for your support

I have identified that the problem is really in the indicator and not in the genetic algorithm.
Also identified the critical parts, I am attaching please take a look and tell me if I'm doing something wrong
I researched within this forum and found hundreds of problems related to memory and almost all unsolved, it really is a serious problem.

[NinjaTrader_Jesse]

Hello,

Thank you for the reply.

Being that this is out of context, I would not be able to really say if anything is wrong in the provided sample. All I can really tell from what was provided is that you are using some conditions based on various indicators and variables.

If you have identified this as the region causing the problem, you can start debugging this region as needed. You can try removing logic to pinpoint the problem. Once you have located the problem, it may be more apparent as to what is happening, otherwise if possible I would suggest re create that logic in a sample script to better isolate what is happening.

If you can provide a sample of logic that will compile and demonstrate the specific syntax used that causes this, we could look more deeply into it to find the reason.

I look forward to being of further assistance.

[Frasson]

Hi Jesse, thanks for the feedback
I identified the problem.
I built a little program that reproduces in a very simplified way the process that I need to do.
It is attached.
Please check the memory in the windows task manager when running the program. The memory increases and does not decrease further.
The program will run for 1 minute after he stops.
Please help me to find some what's going wrong.

[NinjaTrader_Jesse]

Hello,

Thank you for the sample.

I do see that you are using a loop to create many instances of an indicator which could lead to un expected problems when used in efficiently in this way.

Could you instead try storing an instance of the indicator as shown in this post: http://ninjatrader.com/support/forum...48&postcount=3

Please try this approach in your script and see if this resolves the memory issues.

I Look forward to being of further assistance.

[Frasson]

Hi Jesse, thanks for the quick reply

I understand your code, but in my case I need to change the parameters because I'm developing a genetic algorithm.
How I do???
The code below is what's inside the loop.

gene01=rnd.Next(geneMin01,geneMax01);
gene02=rnd.Next(geneMin02,geneMax02);
gene02 = geneTick02 * gene02;
gene03=rnd.Next(geneMin03,geneMax03);
gene03 = geneTick03 * gene03;
gene04=rnd.Next(geneMin04,geneMax04);
gene05=rnd.Next(geneMin05,geneMax05);
gene06=rnd.Next(geneMin06,geneMax06);
gene07=rnd.Next(geneMin07,geneMax07);
gene07 = geneTick07 * gene07;
gene08=rnd.Next(geneMin08,geneMax08);
gene09=rnd.Next(geneMin09,geneMax09);
gene10=rnd.Next(geneMin10,geneMax10);
gene11=rnd.Next(geneMin11,geneMax11);
gene12=rnd.Next(geneMin12,geneMax12);
gene13=rnd.Next(geneMin13,geneMax13);
gene14=rnd.Next(geneMin14,geneMax14);
gene15=rnd.Next(geneMin15,geneMax15);
gene15 = geneTick15 * gene15;
gene16=rnd.Next(geneMin16,geneMax16);
gene17=rnd.Next(geneMin17,geneMax17);

double backtest = EliteEMAFast02(gene01,true,gene02,gene03,gene04,ge ne05,gene06,
gene07,gene08,gene09,gene10,gene11,gene12,gene13,g ene14,gene15,gene16,
gene17, false, false, false, false).ProfitTradeTotalReal[0];

[NinjaTrader_Jesse]

Hello,

The only way to take advantage of the indicator caching would be when the indicator is called with the same exact parameters. Each new set of parameters would create a new instance reference and it would remain until the GC collects and disposes of the objects. In this case this would occur due to the large number of iterations creating a large number of instances of new parameter sets.


I look forward to being of further assistance.

[Frasson]

Hi thanks for reply

Well.....in short I have no way out

When the system call the GC.Collector?

If some times I put the original parameters, the system will call the GC.Collector?

[NinjaTrader_Matthew]

We allow the runtime environment to determine when the GC will run. There are some conditions listed from MSDN, but in most cases it is unpredictable when GC will try to reclaim memory.

https://msdn.microsoft.com/en-us/library/ee787088(v=vs.110).aspx

Conditions for a garbage collection
Garbage collection occurs when one of the following conditions is true:

• The system has low physical memory.
• The memory that is used by allocated objects on the managed heap surpasses an acceptable threshold. This threshold is continuously adjusted as the process runs.
• The GC.Collect method is called. In almost all cases, you do not have to call this method, because the garbage collector runs continuously. This method is primarily used for unique situations and testing.

It is generally bad practice to try and force the GC.Collect() to run and there is no guarantee that memory which you may deem ready to be be released will actually be released at any specific time.

[koganam]

Hi Jesse, thanks for the quick reply

I understand your code, but in my case I need to change the parameters because I'm developing a genetic algorithm.
How I do???
The code below is what's inside the loop.

gene01=rnd.Next(geneMin01,geneMax01);
gene02=rnd.Next(geneMin02,geneMax02);
gene02 = geneTick02 * gene02;
gene03=rnd.Next(geneMin03,geneMax03);
gene03 = geneTick03 * gene03;
gene04=rnd.Next(geneMin04,geneMax04);
gene05=rnd.Next(geneMin05,geneMax05);
gene06=rnd.Next(geneMin06,geneMax06);
gene07=rnd.Next(geneMin07,geneMax07);
gene07 = geneTick07 * gene07;
gene08=rnd.Next(geneMin08,geneMax08);
gene09=rnd.Next(geneMin09,geneMax09);
gene10=rnd.Next(geneMin10,geneMax10);
gene11=rnd.Next(geneMin11,geneMax11);
gene12=rnd.Next(geneMin12,geneMax12);
gene13=rnd.Next(geneMin13,geneMax13);
gene14=rnd.Next(geneMin14,geneMax14);
gene15=rnd.Next(geneMin15,geneMax15);
gene15 = geneTick15 * gene15;
gene16=rnd.Next(geneMin16,geneMax16);
gene17=rnd.Next(geneMin17,geneMax17);

double backtest = EliteEMAFast02(gene01,true,gene02,gene03,gene04,ge ne05,gene06,
gene07,gene08,gene09,gene10,gene11,gene12,gene13,g ene14,gene15,gene16,
gene17, false, false, false, false).ProfitTradeTotalReal[0];

Dispose of EliteEMAFast02 before you make the next call to it.

[Frasson]

Thanks Jesse

Hi koganam

I think I already tried to Dispose before, but I will try again
Thanks for the tip