Resource release/disposal problem for Strategy Analyzer and Market Analyzer

[sajosh]

Hi NT,

NT is awesome.

I think there may be a problem with resource release/disposal for Strategy Analyzer and Market Analyzer on NT8.0.14.2.

SA will not release resources when the SA window is closed nor when the workspace is closed nor when SA is re run. Ideally SA should release (or reuse) all resources when re-run, definately when the SA is closed and most definately when the workspace is closed. However, the only way to release is to close out of NT.

Try it ... open a new workspace with a new SA then use Sample MA Crossover strategy on 15 minute data running back several years (the more data the more evident the resource problem), record NT's memory usage in Task Manager, run the strategy, record the memory again, run the strategy again, record the memory again. You'll notice that the memory usage doubled even though you're running the same strategy with the same data and prior results should be deleted/released/disposed. Now shut the SA, you'll notice that the memory is not released meaning the memory allocation is the same in the task manager. Now shut the workspace, again memory not released. Close NT, resources released of course.


Second similar problem is with MA. It does release resources when you close the MA or the workspace, which is as expected. However, the MA takes just as long to release memory as it does to run the MA, which is a problem. My MA takes 30 minutes to run, then when I exit the MA or workspace or NT it takes 30 minutes to close any one of those. Try it on any new MA with tons of data load and calcs.

So I hope this helps you improve NT.

[NinjaTrader_JoshG]

Hello sajosh,
Thanks for your post.

The way this was explained to me is that this is a product of .NET garbage collection. Because of the way that collection process works it prevents us from disposing of certain resources until the platform is completely shut down. This means that if we are to run several backtests in a row it would be expected for it to lock up resources until the next time you run NinjaTrader.

Incidentally I was able to get a good chunk of memory back by closing the workspace and opening a new one.

Again, thanks for reporting this and let me know if there is anything else I can do to assist. I will leave this post open for anyone else that wants to comment.

[sajosh]

Hi JoshG,


Nice of you to ask them so quickly.


Why does the memory reduce/release for MA window or MA workspace but not for SA window or SA workspace?



Was your remark for MA or SA? ... "Incidentally I was able to get a good chunk of memory back by closing the workspace and opening a new one."

[NinjaTrader_JoshG]

Not a problem. It looks like this is something they have looked at a couple of times previously.

Why does the memory reduce/release for MA window or MA workspace but not for SA window or SA workspace?

I did not see the same type of memory usage with the MA but I would assume the main difference is the resources that are being used.

Was your remark for MA or SA? ... "Incidentally I was able to get a good chunk of memory back by closing the workspace and opening a new one."

This was in regards to the SA

[sajosh]

What are the differences? ... "main difference is the resources that are being used."




So how did you "get a good chunk of memory back by closing the workspace" for SA? For me, closing the SA workspace does not release. Right now I have to exit NT completely after a few SA runs.

[NinjaTrader_JoshG]

I am not extremely familiar with resource management so I cannot answer that question. I was only reporting what I experienced. What I was told by QA is that you should expect to need to shut down NinjaTrader to get those resources back. Sounds like what you are experiencing is expected.

I know this is something they would like to improve but unfortunately I do not have any specific information on that or an ETA.

[sajosh]

NT7 didn't have this SA resource release problem. I was able to rerun many many times without ever shutting down NT. Try it out yourself, you'll see.

told by QA is that you should expect to need to shut down NinjaTrader to get those resources back

What do the developers think?

I am not extremely familiar with resource management so I cannot answer that question.

So how did you "get a good chunk of memory back by closing the workspace" for SA?

[NinjaTrader_JoshG]

What do the developers think?

Their answer will be the same. QA works very closely with them.

So how did you "get a good chunk of memory back by closing the workspace" for SA?

I am not sure. QA didn't say that was a fix. I was simply telling you that is what I experienced. I am not guaranteeing that to be the case for anyone else that tries it and I would need to test further to determine if that is a consistent result. My guess is that it's not.

[sajosh]

Did you get to try it in NT7?

[NinjaTrader_JoshG]

Yes I tried it. Its tough to compare the two though because they are built differently from the ground up.

[sajosh]

Oh good. Does NT7 have the resource release/disposal problem for you?


I understand they are different but NT8 should be better right? I'm simply trying to help NT8 get better.

[sajosh]

Hi JoshG, Does NT7 have the resource release/disposal problem for you?

[NinjaTrader_JoshG]

I see that it is handled differently. QA is tracking your concern. Do you have an additional suggestion you want me to pass along?

[sajosh]

Great. To me, it seems like unwanted behavior (to have to restart a program to release memory). Until then ...



Is there any way to manually release all resources (or specifically data series) on State.Terminated or a state prior to that?

[NinjaTrader_JoshG]

I agree it is unwanted. However, at this time there simply is no way to force .NET garbage collection in the Strategy Analyzer to free up those resources.