Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Frying memory

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Frying memory

    Hi Support Team
    Summary of my problem:
    I developed an indicator that works as a strategy (let's call ELTTS). It has 17 inputs that can be varied (setups) and two outputs. One output is a signal -1 or 0 or 1 that is to use for automation the other output is its own backtest (the strategy profit in the data range defined in the graph). So far so good, works great.
    However as are many input set-ups it takes a lot of time to adjust the best set-up manually. To solve this problem I developed another indicator that tracks the best setups "to ELTTS" using "genetic algorithms". Everything is good so far.
    During development I found that the NinjaTrader gets lost because of the "large memory consumption". Being that I was careful to always clean arrays, lists and arrays in addition to always break the references of the variables in memory (always insert (int) or (double) in front).
    I verified that I can not leave the algorithm naturally converge because it generates innumerable loops and the NT can not take (consume all memory and locks).
    Because of this I put a timer in the system, eg I set to 100,000 backtests and when it reaches "10 min processing" I force stopping the process. The resulting backtests the number depends on the size of the database. In this example got make 1000 backtests in 10 minutes for a small size base. When the processing time exceeds 10 min "the system stops" and gives me the results of the best tracked set-ups.
    I identified that if I run the genetic algorithm immediately after starting the NinjaTrader the system promptly gives me the results in 10 minutes. However if after run once time the algorithm and after I rerun the algorithm genetic a NinjaTrader frie my memory and takes a lot longer than 10min to give me the answer and sometimes not respond.
    I wonder what's going on? And if you have any tips to resolve this the problem of NinjaTrader memory fryer. kkkk
    Another question
    When i put values in the DataSeries that way myDataSerie.Set(value) is diferent than myDataSerie[0] = value?
    Thank you
    Last edited by Frasson; 11-20-2015, 07:35 AM.

    #2
    Hello,

    Thank you for the question.

    Based on the description I would be unsure what may be occurring as I do not know what logic you have used in the files being run.

    The only suggestion I could make based on the provided description would be to try and isolate the code or logic causing this. You could do that by reducing the code until the problem is resolved to find what area of code is potentially leading to this outcome. You would likely need to debug the code to find the answer to this.

    If you are not able to re create the scenario using one of the sample scripts, it is likely something in the logic used in the script so you can use this as a test as well. If you are able to reduce and locate the logic causing the problem, I could look further into a sample of that logic to see what is happening.

    Regarding the .Set, that is an internal method so there are internal processes that occur with that method opposed to using the other version.

    I look forward to being of further assistance.
    JesseNinjaTrader Customer Service

    Comment


      #3
      Hi Jesse thanks for your support

      I have identified that the problem is really in the indicator and not in the genetic algorithm.
      Also identified the critical parts, I am attaching please take a look and tell me if I'm doing something wrong
      I researched within this forum and found hundreds of problems related to memory and almost all unsolved, it really is a serious problem.
      Last edited by Frasson; 11-20-2015, 01:34 PM.

      Comment


        #4
        Hello,

        Thank you for the reply.

        Being that this is out of context, I would not be able to really say if anything is wrong in the provided sample. All I can really tell from what was provided is that you are using some conditions based on various indicators and variables.

        If you have identified this as the region causing the problem, you can start debugging this region as needed. You can try removing logic to pinpoint the problem. Once you have located the problem, it may be more apparent as to what is happening, otherwise if possible I would suggest re create that logic in a sample script to better isolate what is happening.

        If you can provide a sample of logic that will compile and demonstrate the specific syntax used that causes this, we could look more deeply into it to find the reason.

        I look forward to being of further assistance.
        JesseNinjaTrader Customer Service

        Comment


          #5
          Hi Jesse, thanks for the feedback
          I identified the problem.
          I built a little program that reproduces in a very simplified way the process that I need to do.
          It is attached.
          Please check the memory in the windows task manager when running the program. The memory increases and does not decrease further.
          The program will run for 1 minute after he stops.
          Please help me to find some what's going wrong.
          Attached Files
          Last edited by Frasson; 11-20-2015, 04:41 PM.

          Comment


            #6
            Hello,

            Thank you for the sample.

            I do see that you are using a loop to create many instances of an indicator which could lead to un expected problems when used in efficiently in this way.

            Could you instead try storing an instance of the indicator as shown in this post: http://ninjatrader.com/support/forum...48&postcount=3

            Please try this approach in your script and see if this resolves the memory issues.

            I Look forward to being of further assistance.
            JesseNinjaTrader Customer Service

            Comment


              #7
              Hi Jesse, thanks for the quick reply

              I understand your code, but in my case I need to change the parameters because I'm developing a genetic algorithm.
              How I do???
              The code below is what's inside the loop.

              gene01=rnd.Next(geneMin01,geneMax01);
              gene02=rnd.Next(geneMin02,geneMax02);
              gene02 = geneTick02 * gene02;
              gene03=rnd.Next(geneMin03,geneMax03);
              gene03 = geneTick03 * gene03;
              gene04=rnd.Next(geneMin04,geneMax04);
              gene05=rnd.Next(geneMin05,geneMax05);
              gene06=rnd.Next(geneMin06,geneMax06);
              gene07=rnd.Next(geneMin07,geneMax07);
              gene07 = geneTick07 * gene07;
              gene08=rnd.Next(geneMin08,geneMax08);
              gene09=rnd.Next(geneMin09,geneMax09);
              gene10=rnd.Next(geneMin10,geneMax10);
              gene11=rnd.Next(geneMin11,geneMax11);
              gene12=rnd.Next(geneMin12,geneMax12);
              gene13=rnd.Next(geneMin13,geneMax13);
              gene14=rnd.Next(geneMin14,geneMax14);
              gene15=rnd.Next(geneMin15,geneMax15);
              gene15 = geneTick15 * gene15;
              gene16=rnd.Next(geneMin16,geneMax16);
              gene17=rnd.Next(geneMin17,geneMax17);

              double backtest = EliteEMAFast02(gene01,true,gene02,gene03,gene04,ge ne05,gene06,
              gene07,gene08,gene09,gene10,gene11,gene12,gene13,g ene14,gene15,gene16,
              gene17, false, false, false, false).ProfitTradeTotalReal[0];

              Comment


                #8
                Hello,

                The only way to take advantage of the indicator caching would be when the indicator is called with the same exact parameters. Each new set of parameters would create a new instance reference and it would remain until the GC collects and disposes of the objects. In this case this would occur due to the large number of iterations creating a large number of instances of new parameter sets.


                I look forward to being of further assistance.
                JesseNinjaTrader Customer Service

                Comment


                  #9
                  Hi thanks for reply

                  Well.....in short I have no way out

                  When the system call the GC.Collector?

                  If some times I put the original parameters, the system will call the GC.Collector?
                  Last edited by Frasson; 11-23-2015, 05:15 PM.

                  Comment


                    #10
                    We allow the runtime environment to determine when the GC will run. There are some conditions listed from MSDN, but in most cases it is unpredictable when GC will try to reclaim memory.

                    https://msdn.microsoft.com/en-us/lib...v=vs.110).aspx
                    Conditions for a garbage collection
                    Garbage collection occurs when one of the following conditions is true:
                    • The system has low physical memory.
                    • The memory that is used by allocated objects on the managed heap surpasses an acceptable threshold. This threshold is continuously adjusted as the process runs.
                    • The GC.Collect method is called. In almost all cases, you do not have to call this method, because the garbage collector runs continuously. This method is primarily used for unique situations and testing.
                    It is generally bad practice to try and force the GC.Collect() to run and there is no guarantee that memory which you may deem ready to be be released will actually be released at any specific time.
                    MatthewNinjaTrader Product Management

                    Comment


                      #11
                      Originally posted by Frasson View Post
                      Hi Jesse, thanks for the quick reply

                      I understand your code, but in my case I need to change the parameters because I'm developing a genetic algorithm.
                      How I do???
                      The code below is what's inside the loop.

                      gene01=rnd.Next(geneMin01,geneMax01);
                      gene02=rnd.Next(geneMin02,geneMax02);
                      gene02 = geneTick02 * gene02;
                      gene03=rnd.Next(geneMin03,geneMax03);
                      gene03 = geneTick03 * gene03;
                      gene04=rnd.Next(geneMin04,geneMax04);
                      gene05=rnd.Next(geneMin05,geneMax05);
                      gene06=rnd.Next(geneMin06,geneMax06);
                      gene07=rnd.Next(geneMin07,geneMax07);
                      gene07 = geneTick07 * gene07;
                      gene08=rnd.Next(geneMin08,geneMax08);
                      gene09=rnd.Next(geneMin09,geneMax09);
                      gene10=rnd.Next(geneMin10,geneMax10);
                      gene11=rnd.Next(geneMin11,geneMax11);
                      gene12=rnd.Next(geneMin12,geneMax12);
                      gene13=rnd.Next(geneMin13,geneMax13);
                      gene14=rnd.Next(geneMin14,geneMax14);
                      gene15=rnd.Next(geneMin15,geneMax15);
                      gene15 = geneTick15 * gene15;
                      gene16=rnd.Next(geneMin16,geneMax16);
                      gene17=rnd.Next(geneMin17,geneMax17);

                      double backtest = EliteEMAFast02(gene01,true,gene02,gene03,gene04,ge ne05,gene06,
                      gene07,gene08,gene09,gene10,gene11,gene12,gene13,g ene14,gene15,gene16,
                      gene17, false, false, false, false).ProfitTradeTotalReal[0];
                      Dispose of EliteEMAFast02 before you make the next call to it.

                      Comment


                        #12
                        Thanks Jesse

                        Hi koganam

                        I think I already tried to Dispose before, but I will try again
                        Thanks for the tip

                        Comment

                        Latest Posts

                        Collapse

                        Topics Statistics Last Post
                        Started by mballagan, Today, 01:02 PM
                        0 responses
                        3 views
                        0 likes
                        Last Post mballagan  
                        Started by larkmail, Today, 12:30 PM
                        1 response
                        5 views
                        0 likes
                        Last Post NinjaTrader_Kate  
                        Started by omermirza, Today, 12:39 PM
                        0 responses
                        9 views
                        0 likes
                        Last Post omermirza  
                        Started by Danila, Today, 11:08 AM
                        1 response
                        11 views
                        0 likes
                        Last Post NinjaTrader_Jesse  
                        Started by dbida001, Today, 10:31 AM
                        1 response
                        5 views
                        0 likes
                        Last Post NinjaTrader_ChristopherJ  
                        Working...
                        X