Strategy Analyzer memory leak issue/question

[pat9rb]

I'm seeing what looks like significant memory leaks in the strategy analyzer. I can only run a couple of passes before NT stops responding and requires a force quit and restart.

In the screen cap from MSVS below, heap snapshots occurred as follows:



1. Start NT clean (no charts), open Strategy Analyzer (SA) and set up a multi-objective optimization which will run 5 iterations (I have 4 cores) -> Snapshot #1

2. Run SA optimization, wait until complete and GC triggers at least once -> Snapshot #2

3. Run same optimization again -> Snapshot #3

4. Run same optimization again* -> Snapshot #4

5. Close Strategy Analyzer and wait for GC to run at least once -> Snapshot #5

Durring #4, NT regularly stops responding (white-out, system message), and after 5 NT is frequently similarly unresponsive until I terminate its process.

Looking at the heap diffs, it appears the largest increase in memory usage is allocations of new NinjaTrader.NinjaScript.Series<Double> objects.

I've tried running with IsInstantiatedOnEachOptimizationIteration both true and false, and it doesn't appear to make a difference.

Is the strategy or are custom indicators responsible for releasing indicators in State.Terminated? (I haven't seen this in examples and have assumed not.) Any ideas or suggestions for reducing/freeing up heap between passes?

[NinjaTrader_AlanP]

Hello pat9rb,

Thank you for your note.

What version of NinjaTrader are you using? Under Control Center>Help>About?

Are you able to replicate using the SampleSMACross Strategy preloaded with NinjaTrader?

I look forward to your reply.

[pat9rb]

Thanks for your quick reply (and on a weekend!)

What version of NinjaTrader are you using? Under Control Center>Help>About?

8.0.15.1 64-bit

If it helps:

Win 8.1 64-bit v 6.3 build 9600 (this issue also happens on Win 10 64-bit)
.NET 4.7.03062

Are you able to replicate using the SampleSMACross Strategy preloaded with NinjaTrader?

I do get similar results, though not as severe.

Would it help the GC if I null'ed pointers/references to indicators and series during State.Terminated? (I'm somewhat new to C# and .NET)

[NinjaTrader_AlanP]

Hello pat9rb,

Thank you for the information.

We are looking into this and will follow up with you tomorrow.

Thank you for your patience.

[NinjaTrader_AlanP]

Hello pat9rb,

NinjaTrader does not force garbage collection so it’s possible to stack memory when using the analyzer/optimizer. If you are unable to get the SampleMACross strategy to the point where NT becomes unresponsive, this points to your script being the issue.

Debugging using VS for diagnostics is not something we can support.

If you nulled pointers/references to indicators and series in State. Terminated, this may help at the very least to reduce the script down and isolate the problem.

If you nulled these pointers/references, does that change performance?

I look forward to your reply.

[pat9rb]

Hello pat9rb,

NinjaTrader does not force garbage collection so it’s possible to stack memory when using the analyzer/optimizer. If you are unable to get the SampleMACross strategy to the point where NT becomes unresponsive, this points to your script being the issue.

The memory usage pattern seems to indicate iterations on the strategy analyzer will continue to leak until physical memory is filled, at which time NT would become unresponsive. If this is the case, I can test it by simply adding more SMAs to the sample strategy, or running enough iterations to fill memory. The VS profiler seems to indicate this - I can test it out.

Hello pat9rb,
Debugging using VS for diagnostics is not something we can support.

I'm not sure what you mean. I'm not asking for help with VS. Do you guys use a different tool for debugging or runtime profiling? I can try using that tool if it makes troubleshooting the issue easier.

Hello pat9rb,
If you nulled pointers/references to indicators and series in State. Terminated, this may help at the very least to reduce the script down and isolate the problem.

If you nulled these pointers/references, does that change performance?

Let me create a version of the SampleSMACross with more built-in indicators and test it. That'll eliminate any of the custom indicators I've created. I'll report back what I find, but please let me know if you guys use a different tool for detecting memory leaks and I'll see if I can use it instead of MSVS.

[NinjaTrader_AlanP]

Hello pat9rb,

If you are doing things which intentionally stacking up memory, you will eventually run out of memory.

GC is not forced so the optimizer is one place where memory can really add up which would not be the case in realtime or any other mode since you load it once.

Regarding debugging in VS, what I had meant to say was that we would be unable to use your VS profiling information toward debugging, we can however use the platform its self to test various situations. If you are having a situation where the script you use is using a lot of memory, you would want to reduce the script to find what objects are specifically doing that to see if it can be avoided or reduced.

If your script has drawing objects, you may want to add a bool which would allow you to disable them so that when you test in the strategy analyzer. This will help with performance. I’ve attached a sample on how to do this.

A resource monitor should be available soon, if you follow up to this thread next week I should be able to give you more information.

[pat9rb]

If you are doing things which intentionally stacking up memory, you will eventually run out of memory.

I haven't intentionally tried to cause leaks - just added indicators and series to simulate a more robust strategy.

Thanks for the sample code. I do draw some stuff on the chart in the strat so I'll check that out.

GC is not forced so the optimizer is one place where memory can really add up which would not be the case in realtime or any other mode since you load it once.

The GC looks like it's firing (the yellow triangles in the VS memory view) it's just not freeing up memory. (My guess would be somewhere NT is still holding references in a list or array to a bunch of Series<double> after the SA is done with the data.)

I did have something interesting happen though. I was working up a debugging example from the SampleMACross and during one session I experienced the GC freeing memory after SA passes as expected. I'm not sure why yet. I did have some strategies running in sim mode, so that may be related. I need to do more experimenting. Are there any reference data sets that are only scrubbed if there's an active connection?

Two quick items:

- Is there a way to limit thread count on the strategy analyzer? It appears unable to run if two or more passes have additional data series loaded ( AddDataSeries() ) with different parameters.

- The indicator ROC isn't protected for division by zero.

A resource monitor should be available soon, if you follow up to this thread next week I should be able to give you more information.

Cool! I look forward to checking it out.

[NinjaTrader_AlanP]

Hello pat9rb,

You could see Manually resetting class level variables to take advantage of Strategy Analyzer optimizer performance benefits, for how you could reset various objects which may improve performance,

https://ninjatrader.com/support/helpGuides/nt8/en-us/?isinstantiatedoneachoptimizationiteration.htm

Are there any reference data sets that are only scrubbed if there's an active connection?

Could you provide more information on what you mean by scrubbed?

Is there a way to limit thread count on the strategy analyzer?

There is not but I will submit a feature request for this.

The indicator ROC isn't protected for division by zero.

When I apply the ROC indicator to a chart and try passing a 0 for the period, period is changed to 1. Could you please provide more information on what you mean by protected?

I look forward to your reply.