Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Memory Leak in Strategy Analyzer

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #16
    Hello everyone,

    There seems to be a bit of confusion for what is happening here.

    Yes optimizations use a significant amount of memory and yes, the memory is not decommitted right away.

    NinjaTrader 7 had a way of forcing garbage collection and NinjaTrader 8 just lets .NET handle it.

    Peter, when you wrote in over email, you gave a us a video showing your test with the SampleMACrossover strategy where it used a large amount of tick data to build 200 volume bars. That will have a noticeable impact on memory which we do see in your video. I performed this test, and then I continued testing to see if the memory was in fact not getting decommitted. After completing your test, I downloaded the same amount of data for ES 06-20 and before I ran the same optimization backtest on that data, the memory was decommitted. This tells me that .NET garbage collection is working and there is not a memory leak.

    Something else to consider with optimization backtests: By default NinjaTrader will create new instances of the strategy for each optimization iteration. A strategy can be programmed to use IsInstantiatedOnEachOptimizationIteration=false (class level variables will need to be reset in State.DataLoaded) and the optimizer will not create new instances of the strategy for each optimization iteration. The additional optimization iterations will increase the amount of memory utilized, and will increase the the time needed to complete the backtest.

    AI Generate strategies do not generate code that is compliant with IsInstantiatedOnEachOptimizationIteration=false, so AI Generate runs will carry a large processing time in nature.

    SampleMACrossover utilizes IsInstantiatedOnEachOptimizationIteration=false, but I thought this would be worth mentioning for others running optimization backtests.

    IsInstantiatedOnEachOptimizationIteration - https://ninjatrader.com/support/help...niteration.htm

    Peter, if there is a test similar to the video provided where we can observe that memory is not decommitting after a period of time, we can look into that.

    Otherwise, what you would be looking for is to have NinjaTrader implement a way to force garbage collection. We do have a feature request tracking interest behind this with the ticket ID SFT-2111. Let me know if you would like us to add a vote on your behalf.

    I look forward to assisting.
    Last edited by NinjaTrader_Jim; 04-20-2020, 12:24 PM.
    JimNinjaTrader Customer Service

    Comment


      #17
      Ok, yes please add my vote for the ticket ID SFT-2111 to have NinjaTrader implement a way to force garbage collection.

      Thank you for looking into this and explaining it.

      Anyone else who wants this feature please add your vote to this thread.

      Comment


        #18
        Click image for larger version

Name:	Resource exhaustion.PNG
Views:	280
Size:	1.07 MB
ID:	1102158
        I am getting resource exhaustion and ninja hang while doing optimization in Strategy Analyzer still using volume bars.

        This issue has not been resolved!

        I have upgraded to 128 GB DDR4 RAM.

        Ninja still managed to use all of the memory plus exhausted the virtual memory.

        I get a resource exhaustion error. There is also an error with NET UltraView.

        I have included the error message of NET Framework Ultraview.
        There are others of ninja application hang and resource exhaustion error which I will attach in next support emails.

        We have already gone through all of your safe mode steps before and support has verified this is an issue in the previous post. Your response previously was for me to vote for a feature request to force garbage collection, which I did. I think that is a ridiculous response to an issue that you acknowledge.

        How will voting for a feature fix the functional problem? Seriously.





        Attached Files

        Comment


          #19
          I understand this is a garbage collection issue. Here are some specific inputs that seem to affect the resource exhaustion due to limitations of the garbage collection within NET Framework:

          1. Using volume bars
          2. Multiple runs. Exhaustion may occur on the 3rd or 4th run. Do I need to close the strategy analyzer after each run or close the workspace as one user suggests after each run?
          3. Optimizing data series

          For other user's sake how does one effectively work around the limitations of the NET Framework's garbage collection limitations?

          Comment


            #20
            Hello peterslane1,

            You wrote in over email and brought up an issue where resources are not freed when using Optimize On Data Series and using Volume bars. Our Quality Assurance team is investigating that further.

            We will continue to work with you over email and will post any necessary updates here.
            JimNinjaTrader Customer Service

            Comment


              #21
              Originally posted by peterslane1 View Post
              Ok, yes please add my vote for the ticket ID SFT-2111 to have NinjaTrader implement a way to force garbage collection.

              Thank you for looking into this and explaining it.

              Anyone else who wants this feature please add your vote to this thread.
              Its now 18 months later and still not working! Anything with tick data or even minute data with lots of data point to analyse on Strategy Analyzer is very very frustrating!! Almost useless, will this ever get fixed ?

              For what its worth, YES PLEASE add my vote to ticket ID SFT-2111 !!!

              Comment


                #22
                Hello Jzgmma,

                We have added a vote on your behalf for improved garbage collection.

                However, the issue is not so much a NinjaTrader issue so much as optimizations are extremely resource intensive.

                Optimize On Data Series can also be extremely resource intensive in addition to regular optimizations.

                I'm including a video that shows how resources may be quickly consumed even with a simple IsInstantiatedOnEachOptimization=false strategy like the SampleMACrossover strategy. This can help illustrate where memory consumptions goes in an optimization. Optimizations that create more trades will have higher memory consumption, and it will be best to configure optimizations that are manageable.

                Understanding Memory utilization in the Strategy Analyzer

                I have created a video demonstrating how the SampleMACrossover strategy, a simple strategy that utilizes IsInStantiatedOnEachOptimizationIteration=false for efficiency, can still quickly utilize memory resources.

                Demo — https://drive.google.com/file/d/15pz...w?usp=drivesdk

                We should consider the following for memory consumption:

                Data * Strategy resources * Number of optimization iterations * Number of trades * Keep best # of results.

                As we can see there are a number of factors that are involved and memory utilization can climb very quickly depending on a few of these factors. Once memory gets maxed out, we can experience short freezes where memory is decommitted, stored to disk, and then new resources are committed before the backtest is resumed.

                We can easily control the number of iterations involved and we can also consider writing our strategies to use IsInstantiatedOnEachOptimizationIteration = false; (Which requires that we reset class level variables in State.DataLoaded.)

                IsInstantiatedOnEachOptimizationIteration — https://ninjatrader.com/support/help...niteration.htm

                Optimization Tips — https://ninjatrader.com/support/help...ionPerformance

                Walk Forward Optimization — https://ninjatrader.com/support/help...ss_metrics.htm

                Genetic Optimization — https://ninjatrader.com/support/help..._algorithm.htm


                JimNinjaTrader Customer Service

                Comment


                  #23
                  Was this ever addressed or fixed? I'm on NT 8.0.26 and just ran the SampleMACrossOver and still see my ram fly through the roof.
                  My own custom strategy does the same. We really need a way to make sure the RAM is cleared out in a timely manner.

                  Comment


                    #24
                    Hello DieSlower,

                    The Strategy Analyzer keeps trades in memory. On systems with enough memory this makes optimizations very fast.

                    On systems with less memory, the number of trades, what is used by the strategy, and the data involved, can reach the PC's memory limits where we then see stalls and stutters.

                    What you can do in the near term:

                    1. Reduce Keep Best # of Results
                    3. If using Optimize Data Series, consider disabling to run smaller faster tests.
                    2. Program the strategy to use IsInstantiedOnEachOptimizationIteration = false (involves resetting class level variables in State.DataLoaded)
                    3. If Trade information is not needed, consider setting IncludeTradeHistoryInBacktest to false.
                    4. Consider using Genetic Optimizations to test less iterations.

                    In regards to SFT-2111 (improved garbage collection/better memory management in the Strategy Analyzer) I have added your vote.

                    Otherwise, an immediate solution would be to test on a system with more memory.

                    Recently, we noted some optimizations taking up to 24 GB of memory with the above factors set abusively. If you are able to test on machines with different amounts of memory available, you may find a sweet spot for having enough for your needs without going overboard.
                    JimNinjaTrader Customer Service

                    Comment


                      #25
                      Hello,

                      My system has 128GB of RAM, so it should have enough for this. I use about a week of data for my tests, but I do have a large number of parameters.
                      I keep 10 best results, so that should not be a problem, my iterations number in the millions though, so I need a way to let unused memory go.

                      I cant reliably use IsInstantiedOnEachOptimizationIteration because of the complexity of the simulation. Otherwise I would gladly use that option, although even with that on, I used to run into memory problems.

                      Thank you for adding my vote to the ticket. In the meantime, do you ahve some code example of how we can manually reduce memory after every strategy termination?



                      Comment


                        #26
                        Hello DieSlower,

                        I do not have any other recommendations or sample code to provide to force garbage collection or to change the way memory is handled in NinjaTrader.

                        The juice may be worth the squeeze to make the strategy IsInstantiedOnEachOptimizationIteration = false compliant. You could test with it set to false, and you should see faster results (albeit incorrect if class level variables are not reset in State.DataLoaded.)
                        JimNinjaTrader Customer Service

                        Comment

                        Latest Posts

                        Collapse

                        Topics Statistics Last Post
                        Started by Mindset, Today, 03:08 AM
                        0 responses
                        4 views
                        0 likes
                        Last Post Mindset
                        by Mindset
                         
                        Started by Genesis2021, 06-24-2022, 05:19 PM
                        1 response
                        17 views
                        0 likes
                        Last Post NinjaTrader_Jason  
                        Started by thommo, 06-24-2022, 02:16 PM
                        1 response
                        25 views
                        0 likes
                        Last Post NinjaTrader_Jason  
                        Started by sgordet, 06-24-2022, 01:30 PM
                        3 responses
                        30 views
                        0 likes
                        Last Post NinjaTrader_Jason  
                        Started by Alpha2Omega007, Yesterday, 10:39 PM
                        0 responses
                        13 views
                        0 likes
                        Last Post Alpha2Omega007  
                        Working...
                        X