Strategy Optmizer Memory Leak

[kevinenergy]

Hi there,

Attached is a Zip file containing a MovingAverageCrossover Strategy along with a simple indicator which demonstrates the Memory Leak issue with the Optimizer, which I have previously reported.

I loaded Ninjatrader from a cold start. Prior to starting the Optimizer it was using 408Mb.

After 600 Optimizer iterations on "MA Cross Mem Leak" (Fast:200 Slow:200 / ES 12-18 / 1 min / 3-Dec-18 to 7-Dec-18) Memory usage was 4.71Gb and the optimization process was getting slower and slower.

I generated the following Dump file:

https://1drv.ms/f/s!Ajlj7dTld4OYhONEh8KLIp9MHCdcVg

This indicates 600 Strategy Instances, 600 Indicator instances, 3 Million Line Drawing Objects and 7 Million Chart Anchors all allocated on the heap.

The memory leaks seems to be caused by a NinjascriptDrawingToolDictionary instance (optimizerInstanceDrawnTools). This type has a Dictionary of entries. The Key of the Dictionary is the indicator which drew the line, and value is a dictionary of drawn objects. The indicator has a reference to the strategy.

Note that I'm using IsInstantiatedOnEachOptimizationIteration = true

[NinjaTrader_Jesse]

Hello kevinenergy,

Thank you for the post.

I reviewed the sample and I do see that you are using a line in the sample. We have had other posts about this in the past :

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/109990-increase-memory-usage-during-optimization-for-indicators-with-draw?p=844811#post844811

This is generally expected in a backtest/optimization situation as the garbage collection is not being forced and many objects will be created. NinjaTrader_Jim provided an explanation from QA in this thread along with some ideas on working around this problem.

I could suggest adding a switch in to turn off drawing objects during a backtest as the other thread recommends. I could also put in a feature request for this, but this would currently be expected to occur in this use case.

I look forward to being of further assistance.

[kevinenergy]

Jesse,

.NET Garbage collection is automatic, it does not need to be "forced" in any way. That being said, to prove the point I have created a modified version of Ninjatrader's Default Optimizer, which once-per-minute allows a single thread to choose itself to force garbage collection on the heap. Unfortunately The GC-enabled optimizer also runs out of memory.

I've also attached the Ninjascript output from this, which clearly shows garbage collection occurring, with about 15Mb being freed from the heap per miniute with another 100Mb leaking per minute.

I've uploaded another dump file from the Garbage-collection-enabled optimizer here: https://1drv.ms/f/s!Ajlj7dTld4OYhONGwKQpliV4YOOEag

and have attached more memory profiling data from the garbage-collection-enabled optimizer-run to this thread also.

Ninjatrader's optimizer has, without question, got a memory leak. That is a categorical fact.
If your development team say otherwise after seeing this information, then I'm afraid to say that they are either incompetent or indifferent.

And yes, of course, not drawing things will obviously create less objects, and yes, this will cause LESS memory to leak. But this suggestion is analogous to using less water to solve the problem of a hole in your bucket. Using less water doesn't fix the hole.

[NinjaTrader_Jesse]

Hello kevinenergy,

Thank you for the reply.

You are correct, the garbage collection is automatic and does not need to be forced, to clarify I had mentioned this as the platform does not control the garbage collection, .NET does. Even in cases where you do force garbage collection, this would not guarantee that the objects in question are collected at that time so you still may run out of memory when doing this specifically. The only solution to this would be to reduce the number of objects being created in memory.

The + CurrentBar tag being used indicates that if this line was to be called on each bar then each bar would have a unique drawing object created. I don't know how many bars this is in your specific test but very likely there would be thousands of drawing objects on a single backtest not to mention that would be multiplied by the number of permutations of an optimization which can result in a lot of memory used.

I can put in a feature request here for development to review this further but this test would be considered expected as many drawing objects are being used to stack the memory. Your test only demonstrates stacking memory up, was there a specific reason that brought your question up that I could relay to development for consideration? The current test does not demonstrate any specific reason that the objects would be needed in the backtest so my suggestion would still be to omit them if they are purely visual. The platform does disable the rendering of the objects you create in a backtest but that does not get around having to create an instance of every object in memory.

Please let me know if I may be of further assistance.

[kevinenergy]

Jesse,

I'm using a number of third party indicators that draw various objects. I need to go through these and conditionally remove all Drawing calls (I am in the process of doing this).

Here's a few reasons why I think this needs to be fixed:

- Ninjatrader has a commercial ecosystem - what if I was using an indicator which I don't have the source code for? I'd have no way to remove the drawing objects and would be unable to use commercial indicators like this in my strategy if I wanted to make use of the strategy analyzer.

- I've configured the strategy analyzer to keep only the best 10 results. Therefore, when it discards poorly-performing strategy-runs, why are the associated drawing objects not discarded also? Unless the drawing objects are associated with the top-10 then they will never again be seen or used. So why keep them? All they do is consume memory and make the strategy analyzer get slower and slower until it runs out of memory and the whole platform crashes. They serve no purpose if they're not in the top-10. Lets face it this is not "by design" its a design oversight. As long as the draw objects exist in that collection they will never be marked for garbage collection and removed from memory, and since they reference the indicator and strategies; the old strategy/indicator instances will never be removed either.

- We are told that the solution is 'Don't use drawing objects' yet Drawing Objects can be very useful in strategy development such as for visualizing order positions (e.g. target orders that never hit, or the progression of a trailing stop), support and resistance lines highlighted by a strategy, signals, status text. Its very useful to be able to see these things in the Strategy Analyzer results on the "Chart" section to help with development and design of the strategy rules. However because of this issue we are forced to abandon chart drawings completely and we lose their benefit during strategy development.

- The dictionary that is caching all drawing objects and causing this problem is called "strategyOptimizerDrawnTools" - if the answer to this problem is "don't draw objects", then why does "strategyOptimizerDrawnTools" even exist? Rather than telling us not to draw objects would it not be better to remove it completely?

[kevinenergy]

Incidentally, fixing this memory leak would be as simple as this for one of Ninjatraders engineers:

The pseudo code below basically says: "If the current optimization is not one of the best 10, remove the strategy instance from the DrawingToolDictionary"

// Pseudo Code
var numberOfBestResultsToKeep = 10;
var bestResults = List<Optimization>();


if( currentOptimization.Performance < bestResults.Min( x => x.Performance)) {
      // Actual Code
      optimizerInstanceDrawnTools.entries.Remove(currentOptimization.Strategy);

}

Removal of this reference in the dictionary would allow the Garbage collector to determine that these drawing tools are no longer needed and the garbage collector will free the memory automatically. As long as that reference is there, the garbage collector will determine that they are still needed and will NEVER evict them from memory. And of course, the drawing tools reference the indicator, and the indicator references the strategy - so they never get garbage collected either.

And in case anyone tries to argue that this isn't actually a memory leak, Here's why adding items to a collection in C# but failing to remove them after they're not needed is, in fact, a memory leak:

https://csharp.2000things.com/tag/memory-leak/

"Assuming that we never remove someone from this list, none of the objects added to the list will ever get garbage collected–because we’ll continue to reference them indefinitely. "
--- This is what Ninjatrader's optimizer is doing. Adding items to a collection but failing to remove them after they're not required any more. Therefore Ninjatrader's optimizer has a memory leak.

https://blogs.msdn.microsoft.com/davidklinems/2005/11/16/three-common-causes-of-memory-leaks-in-managed-applications/

"If at least one reference to a managed object remains in the current scope, the Garbage Collector cannot tell if your application is finished with the object. Since, as far as the GC can tell, the object is still in use, it cannot be marked for collection and therefore the memory is not freed."
--- So, internally, Ninjatrader's optimizer has a DrawingToolDictionary Instance. Drawing tools are added to this collection, but even if the strategy is not in the Top 10, the drawing tools are never removed from the DrawingToolDictionary. This is a MEMORY LEAK.

[NinjaTrader_Jesse]

Hello kevinenergy,

Thank you for the additional details, I will provide this to development for further review. The tracking id for this request is: SFT-3302

Please let me know if I may be of additional assistance.

[hiikila]

Any news on this topic? I clearly have the same problem and memory get's eaten up until the whole system stops working. So please FIX it ASAP.

[NinjaTrader_Jesse]

Hello hiikila,

I have updated my prior post with the feature request tracking id. If this is changed in the future you can find this in the change log here: https://ninjatrader.com/support/help...ease_notes.htm

Please let me know if I may be of additional assistance.

[madb123]

was this fixed? im having a memory leak too..and its almost 2024

[NinjaTrader_Jesse]

Hello madb123,

If you are having an actual memory leak (ever increasing memory that does not get collected ever) please start a new post so that we can collect details on that situation to see if the logic you used is the cause. This post is not for a memory leak but is for over utilization of drawing objects. Over using drawing objects is expected to increase memory in optimizations because many tests are run concurrently, that memory will eventually release on its own but it is expected to dramatically increase memory use if you use a lot of drawing objects in your code.