Proper use of Dispose()

[egan857]

Hi, I have a memory leak and am trying to resolve it. Memory increases at a rapid and constant rate. It starts the day at 1GB and is above 6GB before half the day is gone. I have tried variations of Dispose() and C# garbage collection (GC.Collect(); ), but those cannot be run by themselves to remove all the garbage because that maxes out my CPU for a few minutes. Instead, I need to selectively dispose or do garbage collection. The question is, how?

So, for example, let's say I have the following two lines of code that execute once a minute:

stkADX = ADX(14)[0];
ADX(14).Dispose();

Does this release any space? Did I initially allocate enough space to hold the values of ADX for all the bars in my chart, or did I just allocate the most recent value? When the Dispose statement is executed, does it get rid of that space?

Similarly, I have a couple of statements like this:

low10Day = MIN(Lows[1],10)[0];
MIN(Lows[1],10).Dispose();

Even if the answer to the previous question is "yes, it allocated lots of space and yes you are disposing it", does the MIN function that looks at 10 bars eat up enough space to store 10 values, and am I disposing that space?

Finally, how would I get rid of historical data? When the program starts, I loop through the most recent 3000 1-minute bars to do a couple of calculations. After those calculations are done, I never need to look back more than 60 bars, and even then I only do it once a minute. Are all 3000 bars loaded into memory, and if so is there a way I can release 2940 of them?

[NinjaTrader_AdamP]

egan,

Generally when you call an indicator in NinjaTrader it adds the object for you in a way. The dispose method just removes it from memory. I am actually not sure if there is a default dispose option in the Indicator class so it may not be doing anything here.

You can also implement your own dispose methods : http://msdn.microsoft.com/en-us/library/fs2xkftw.aspx

If you want to keep closer track of your objects, you can use something like this as a reference for creating an indicatorBase object : http://www.ninjatrader.com/support/f...d=4&linkid=535

If you do that you can then dispose of it at your leisure and also be sure its removed from memory. Please note that there may be increased overhead here as you create objects and dispose of them.

If you create an indicator, it actually stores up to an infinite number of elements for its plots, however in practice it stores as many elements in a dataseries plot as there are bars on a chart (also a computer can't store an infinite number of elements).

These bars are stored in memory if they are stored on your chart. If you want to limit the size of any dataseries, you can use something like MaximumBarsLookBack.TwoHundredFiftySix :

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?maximumbarslookback.htm

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?forcemaximumbarslookback256.htm

Please let me know if I may assist further.

[egan857]

Hi AdamP,

Thanks! Actually, I was just reading (and not understanding) that MSDN article a few minutes ago. If I created my own Dispose method, what should be in it that isn't in the default one, so I can get rid of memory used by indicators?

My NinjaScript strategy starts off looping through the most recent 3000 bars one time, and from that point on it only needs to go back 60. Is it possible to set MaximumBarsLookBack to Infinity, then change it to 256 after my one-time calculation, AND release the memory used by all those 3000 bars? I can't do any of the following to remove chart data:

Lows[1][1].Dispose();
Lows[1].Dispose();
Lows.Dispose();
CurrentBars[1].Dispose();
(I'm new to C# - only since January, and only with Ninja scripts).

I think what I'm most confused about is: what's getting allocated when I do certain things (indicators like ADX, or functions like MIN and MAX), and does it get released with Dispose? It sounds like you're saying that ADX(14).Dispose(); is likely not releasing anything, and that seems to be the case based on my tests.

[NinjaTrader_AdamP]

egan,

Before going that route, I would suggest trying another thing. There are some under-the-hood items where the dispose() method may not be applicable for the way you are calling indicators.

Try to instantiate your own indicator objects, something like :

IndicatorBase myindicator = new Indicator()

Then try using dispose. Apparently, indicators do have a Dispose() method according to one of our developers but it may not work like you expected calling indicators like : EMA(15).Dispose().

Unfortunately we are in the unsupported realm here so I am not able to give much further assistance there.

As far as your other question about data series. The reason some data series have infinite elements set is that they are plots. A plot must have this in order to display on your chart further than 256 bars back. If you have no need to plot something you are storing, I would instead suggest using your own data structure for storage, like an array or something. Then you can just set it to 3000 as a size initially, do whatever you need to do, then remove it from memory and reset its size later to something smaller.

Another thing I just thought of here, if you have a lot of messages being logged in your Log tab it can increase your memory usage.

Typically I haven't seen most indicators causing an issue with memory unless they are quite memory intensive or there are a lot of charts open however, so it may be something else going on. Is it primarily NinjaTrader using up a lot of memory on your system?

[egan857]

This isn't an indicator, it's a strategy. I don't have my own indicator objects within the strategy, but I do use some of the standard indicators like ADX, Bollinger, Keltner, and ParabolicSAR. I'm not plotting charts and I'm not storing large arrays. The situation I'm trying to solve is that I'm using a FOR loop to walk through 3000 bars, and incrementing one variable as I walk through the loop. I don't have a 3000-element array that needs to be released, but my strategy has probably loaded information about the 3000 bars in memory and probably won't ever release it.

[NinjaTrader_AdamP]

egan,

Could you give an example of the section that is iterating through 3000 bars? When you attach this to a chart, does your chart have 3000 bars on it?

[egan857]

There are no charts. This is a NinjaScript strategy, not an indicator. I'm looping through historical bars to accumulate one variable, which is then used (among many other criteria) to decide whether to buy or sell the stock.

[NinjaTrader_AdamP]

egan,

Strategies are still constrained by the amount of historical data available on the chart. You can read a little here :

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?barsrequired2.htm

This means that X bars must load on a chart before it allows the OnBarUpdate() method to execute.

So have you only used this in the Strategy Analyzer? Over how long are you running your strategy?

NinjaTrader would release the memory as soon as a single optimization iteration or backtest is run. Optimizations run multi-threaded so it may increase memory usage if for example you have 8 cores and each run is using a lot of memory. However, you have said that you are only storing your calculation in a single variable. I really don't think that your iterating over 3000 bars is causing this issue here, however a quick test would be to change this to something like 500 and see if it makes any difference.

I would suggest printing out the "CurrentBar" when you run a single backtest to get some idea how many bars you have accessible when you first start running your strategy.

If you have a "toy" strategy that exhibits this behavior it would be beneficial for me to test as well, and also to review the code. I understand a reluctance to post code publicly.

[egan857]

Hi AdamP,

This is a live strategy. I have never backtested this, run it through the strategy analyzer, or the optimizer. It's a real strategy running real-time trading real shares. I'm a quant, I have lots of experience with autotrading systems, and I don't believe that backtesting improves results. I start 200 copies of this program every morning and disable all of them when I run out of memory a few hours later, and then I shut down NinjaTrader, clear out everything in my cache directories, start NT, and start the strategies again until they run out of memory again.

When I start the strategy, one of the parameters is to load 20 (calendar) days' of history. A minute or two after the program initiates, it has typically loaded around 4500 1-minute bars, so the value of CurrentBars[0] is 4500. Thinner stocks don't have as many 1-minute bars, which is why I loop through 3000 iterations. As soon as the value of CurrentBars[0] is over 3000, I start the loop. If it takes more than 3 minutes to load 3000 bars, the program disables itself.

I am reluctant to post the code publicly, but will be happy to send it you personally. I warn you, though, it's a beast - 8000 lines, but at least it's well documented.

[NinjaTrader_Brett]

Hello,

You sent in your code and we did review it on a similar report on Friday.

Unfortunately with the code being so large we could not pinpoint anything just from that alone and we were able to provide some places to start in that email chain.

We decided that to move forward we would need to start the process of elimination and start removing code segments until the memory problem went away. Then we have isolated the problem and once isolated we would be able to go from there.

Let me know if you have any questions on the process of elimination needed for next step here.

[ldissinger]

Why don't you tell it to load 60 bars and in the OnStartUp method, load the data you need to do the historical calculations as in:

Data.Bars minutes = Data.Bars.GetBars(Bars.Instrument, new Period(PeriodType.Minute, 1, Bars.Period.MarketDataType), Time[0], BarsArray[0].GetTime(Close.Count-1) , (Session)Bars.Session.Clone(), Data.Bars.SplitAdjust, Data.Bars.DividendAdjust);

Only replace Time[0] with a time 3000 bars earlier which if working with minute charts could be calculated.

You can instantiate those indicators for the calculations needed by passing in ticks to that indicator and let it work on those bars.

Leroy

[pls2000]

Maybe I'm butting in where I shouldn't be, but please let me correct a piece of bad information in this thread.

The primary purpose of Dispoase is to release non-managed (by C#/.Net) objects such as windows, files, database connections, etc. It has only a limited use in pure C# objects in particular situations.

In particular, Dispose does not release memory. Only the Garbage Collector can release the memory used by an object. Even GC.Collect is not guaranteed to release memory.

From your description, the problme is not that your program is creating and dropping pure C# objects. It sounds like the objects you are creating are for some reason remainng referenced and therefore not able to be released by GC. Not having seen your code I can't say much more.

I am surprised at the NT developer recommendation to remove segments of code. There are well known tools for diagnosing this kind of problem available both from Microsoft and from other vendors. Looking at counts of referenced objects by type and objects elegible for GC will go a long way to pointing you to your problem.

++PLS

[NinjaTrader_Brett]

This thread is a few years old so I do not know if the poster will respond or not. However a tool that does this is any kind of .net memory profiler. Visual studio 2015 has some memory profile tooling built in now and if you wanted to go third party you could use something like redgate memory profiler.

[NinjaTrader_PatrickH]

Hello leocrespo,

We have created a video on the profiler for you at the following link: http://screencast.com/t/ZQAoWl3Ms

Please let me know if you have any questions.

[NinjaTrader_PatrickH]

It is public so feel free to share.