• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

memory leaks in strategy run by Optimizer?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    memory leaks in strategy run by Optimizer?

    Hi,
    please point me to articles or hints what could be causing memory leaks when my strategy is run in Optimizer?

    Ninja Trader8 starts with roughly 480 MB in RAM, then at each run of Optimizer it adds another 200-300MB consistenly, until my RAM usage is 98% at which point I have to kill the process and start again... this happens after about 10 runs ... what might be causing this?

    IsInstantiatedOnEachOptimizationIteration = true
    changing it to false does not help

    In State.Configure I do add a secondary data-series and two float series for a custom indicator:

    // add minute data series to watch for EOD
    AddDataSeries(BarsPeriodType.Minute, EODPeriod);


    // input data series that AAAAAAAAindicator will use to plot in indicator's panel
    // we supply this data here, as opposed to a standard indicator which uses bar data as input
    IndicatorLong = new Series<double>(this);
    IndicatorShort = new Series<double>(this);

    then I fill up the above with historical data in State.DataLoaded, then update at each OnBarUpdate() with the new bar value:
    (example)
    IndicatorShort[0] = XXXXX.Value[0];
    IndicatorLong[0] = YYYYY.Value[0];

    Should I release/free all this data somehow after each run to avoid this piling up of memory usage?

    thanks in advance,
    Momchil

    #2
    Hi,

    somewhat related, also a problem is that the strategy analyzer requires a whopping 7GB of RAM on my computer without having any test run(latest NT version).
    To be sure that it's not caused by any of my code, I disabled all my indicators and strategies by switching them to mock classes that extend NinjaScriptBase but not Indicator or Strategy so NT doesn't find them.

    My test procedure was:
    1.) open NT -> 160MB with one Market Analyzer running
    2.) open Strategy Analyzer -> 10.5 GB
    3.) SampleMACrossOver optimization -> 7GB

    It was able to free up 3GB, but I'd curious why the analyzer requires a long startup time and RAM. NT managed to fully occupy all my free RAM on previous tests.
    I can only theorize that back testing and optimization don't need to preload a lot of data. If it turns that a probably rarely used feature like the AI needs all these resources would it be possible to split it off from the strategy analyzer?

    I am not sure it's somehow related to the latest version of NT, Maybe I'll give it a try to revert back and will report later if the situation improves.
    However I was able to run some optimizations with the current version earlier so I'd guess it's possibly not caused by new code.

    Comment


      #3
      Hello Momchil,

      Thanks for your post.

      When using IsInstantiatedOnEachOptimizationIteration = false, the strategy must reset all class level variables in State.DataLoaded to ensure that left over values are not used since the script will not be instantiated for each optimization iteration. This would be advised for best performance when backtesting.

      Garbage collection will take place when NinjaTrader needs memory. We should expect that memory will be freed periodically when the strategy is running the optimization and the platform should not crash.

      With regards to factors impacting memory load, heavy usage of lists and similar objects can do this, strategies that create large number of trades can increase this, and using high values for Keep Best # of results will increase the amount of data that is kept in the optimization.

      momchi I would suggest ensuring that your script is compliant with IsInstantiatedOnEachOptimizationIteration = false, and I would suggest approaching this from two angles:

      1. Test to reproduce the issue from a clean start of the platform with workspaces closed so we know we are starting at a clean point and so we can focus on where memory is getting spent. I would suggest then to reduce parts of your strategy code to identify parts that are consuming memory. Keep track of the settings used as well. Between tests, please make sure to restart the platform so we are testing from a clean slate.

      2. Keeping track of the settings used, attempt to reproduce the symptoms using the same settings, but with the SampleMACrossover strategy. This should help to rule out if memory consumption is high for your specific strategy.

      MojoJojo I do not see these symptoms when I open NinjaTrader and then open a Strategy Analyzer. Memory usage stays low until I run an optimization when I take the same steps you are. Could you reach out to me at platformsupport [at] ninjatrader [dot] com so we can further discuss this? I would like to keep this forum thread on topic with momchi's specific inquiry. Your symptoms seem a little different. Please include the text "2588178 Attn Jim" so the ticket gets assigned to me.

      I look forward to assisting.
      JimNinjaTrader Customer Service

      Comment


        #4
        Just a quick and last update.
        For people who performed a lot of optimizations, after deleting the content of %HOMEPATH%\Documents\NinjaTrader 8\strategyanalyzerlogs, everything is back to normal.

        Comment


          #5
          Hi Jim, thank you for the reply -
          as I am working through suggested steps, a question: what states a strategy goes through at the start of a new optimizer step when IsInstantiatedOnEachOptimizationIteration = false?
          In particular these are of interest to me: Configure, DataLoaded and SetDefaults.

          Comment


            #6
            Hello momchi,

            Strategies will process Configure, DataLoaded, and then Historical when a new optimization iteration takes place. We suggest resetting class level variables in State.DataLoaded for a single state to perform these operations since variables based on loaded data could not be reset in State.Configure.

            Let us know if you have any additional questions.
            JimNinjaTrader Customer Service

            Comment

            Latest Posts

            Collapse

            Topics Statistics Last Post
            Started by Dan_D_Trader, Today, 03:29 PM
            0 responses
            1 view
            0 likes
            Last Post Dan_D_Trader  
            Started by Bsmckm, Today, 02:02 PM
            0 responses
            8 views
            0 likes
            Last Post Bsmckm
            by Bsmckm
             
            Started by markyb0y, Today, 01:55 PM
            0 responses
            4 views
            0 likes
            Last Post markyb0y  
            Started by SlickRick, Today, 11:07 AM
            2 responses
            10 views
            0 likes
            Last Post NinjaTrader_ShawnB  
            Started by JTizz, Today, 12:17 PM
            0 responses
            7 views
            0 likes
            Last Post JTizz
            by JTizz
             
            Working...
            X