RAM not being released

[Gains]

Ninja 6.5 beta seems to have a garbage collection issue.
Maybe you guys are aware of this but here is what I observed just in case it might help.

Running on a Vista Home Premium system with 2GB RAM I observed the following RAM usage in the Windows Task Manager.
Ninja running, not connected, no charts =32,000K
Ninja running connected to MB/EFX =40,000K
Opening a workspace with 8 charts and a Market Analyzer with 8 forex pairs =306,000K
Closing the 8 chart/market analyzer workspace leaving nothing open should drop RAM use back down to 40,000K but it actually creeps up to 308,000K used.
Opening the same workspace again with 8 charts and market analyzer and RAM use is 485,000K.
After about 5 minutes of doing nothing it drops to 400,000K, still almost 100,000K higher than the first time the workspace was opened.
After about 10 minutes of running Ninja in the background RAM use has dropped to around 399,650K
Bringing Ninja back to the foreground and RAM use pops back up to 400,000K area.
Closing the 8 chart and 8 pair market analyzer workspace and waiting...
I have only been running the empty default workspace for the past 5 minutes and Ninja is still using 400,000K
Disconnecting from MB/EFX has no effect.
Connecting to Gain raises memory use to 405,000K still with default empty workspace only.
Shut down Ninja and restart and I am back at 32,000K used.
There you are! Hopefully you are already dealing with this issue.

Apart from the above issue is there any chance that RAM usage will be lower in the final release version of 6.5? Right now running an 8 chart workspace with no Market Analyzer takes about 156,000K. I can easily imagine having 8 x 8 chart workspaces for a total of 64 charts. This would take about 1.3GB of RAM space and having tried it I know it chokes my 2GB Vista system.

If RAM use is going to stay in this range has Ninja been tested to run on a 64bit version of Windows or a 32 bit Windows Server version? My understanding is that these are the only flavors of Windows that can utilize 4GB of RAM and higher while XP Pro can only address around 3.3GB.

Thanks for all the hard work.
Gains

[NinjaTrader_Ray]

Will forward this to development.

[NinjaTrader_Dierk]

.NET is a garbage collection system -> NT has no control on when memory no longer in use is given back by .NET runtime to the windows OS.

[Gains]

Ok... reading between the lines of your answer...

You must be saying the .NET framework on my machine is not working right. Right?
Is the case I presented unique or something you are seeing on other machines?

To test this myself I set up Ninja on an XP machine with the same charts and indicators and the RAM usage is much lower, only 100,000K vs 300,000K on my Vista box.
Is Ninja known to use about 3 times as much RAM on Vista machines or is it just my Vista? Is this a Vista "feature"? ;-)

I observed the same memory leak problem on my XP box only it is less of a problem since it only is gobbling up RAM at 1/3 of the rate.
When I shut down my workspace for several minutes the 100,000K was not freed up by Ninja.
When restarting the workspace RAM use went up to 135,000 area but eventually dropped back to about 115,000K, more than what the workspace used the first time I ran it but manageable compared to Vista.

So is it simply my .NET misbehaving? I will look into fixing it.
Or is it just the way things are going to be on Vista machines and to a lesser degree on XP until Microsoft fixes it?
If that is the case and it is not a bug then Vista can not be used to run Ninja if you want to do multiple charts... and I haven't even tested what running multiple strategies will do to it.

I need to know if buying a cheap XP box with 2GB of RAM will be the solution.

One more thought... while .NET does in fact automatically handle the garbage collection task you still - in some cases - have to yell at it to take out the garbage. From a quick Google search...

"The Common Language Runtime (CLR) cannot clean up resources like database connections, window handles, and file handles. Therefore, it is necessary for the developer to provide mechanisms for cleaning up these unmanaged resources. The clean up for this can be implemented in the Finalize method. The Finalize method is implemented as a destructor in C# language. The calling of Finalize is still under the control of the garbage collector.
Usually, you would need a deterministic way to clean up these unmanaged resources like file handles. For example if you have opened a file for reading and you have finished loading the contents of the file into a buffer, you might want to explicitly close the file handle. For this explicit clean up, .NET provides the dispose design pattern."
Quoted from here http://www.builderau.com.au/architec...0282141,00.htm

Thanks.

[NinjaTrader_Dierk]

- if you experience differences between Vista and XP then this might be related to different setups or different .NET management logic. On the latter I suggest you contact MicroSoft for clarification
- you are not experiensing memory leaks (memory is not freed by NT) but that memory is not given back from .NET runtime to windows. This is expected behavior of the .NET runtime. Please contact MicroSoft for details
- triggering the garbage collection will not guarantee that .NET runtime really return memory back to windows
- the info you pasted relates to OS ressourecs and not memory managed by the .NET runtime

[Gains]

Wow

I have so many thoughts about this response that I don't know where to begin...

Let me just keep it simple this time. Can you please answer these two seperate questions which went unaddressed from my first post?

#1
"... is there any chance that RAM usage will be lower in the final release version of 6.5? Right now running an 8 chart workspace with no Market Analyzer takes about 156,000K. I can easily imagine having 8 x 8 chart workspaces for a total of 64 charts. This would take about 1.3GB of RAM space and having tried it I know it chokes my 2GB Vista system. "

#2
"If RAM use is going to stay in this range has Ninja been tested to run on a 64bit version of Windows or a 32 bit Windows Server version? My understanding is that these are the only flavors of Windows that can utilize 4GB of RAM and higher while XP Pro can only address around 3.3GB."

It really would be good to know!

Thanks

[NinjaTrader_Ray]

1) No improvements have been made from the current beta to the expected production release in the area of memory utlitization. This is on our list for improvement in NT7

2) 64 bit windows is not officially supported since we have not tested it. Testing this platform is scheduled for 2008.

[Pipqueen]

Hi Ray and all

I am looking forward for this fix too. I did run 60 charts and does eat the memory on my computer so I know better to be careful until next version

Thank you for the amazing work!
Pipqueen

[Big D]

Even though Ninja is not officially supported on 64 bit versions of Windows (XP 64 or Vista 64), we have been using Ninja (both 6.0 and 6.5 beta) on XP 64 for at least 6 months with almost no problems.

I say almost, because there seems to be a display/refresh slowdown on the DOM in 64 bit. If you compare the DOM on a 32 bit OS with a 64 bit OS, the 64 bit will not refresh as fast or as smoothly. This is perhaps only a problem for a professional daytrader who is used to TT's DOM and can notice the smoothness that every market should move at.

Ray or Dierk: You should really look into this as Ninja is an excellent and cheaper alternative to TT and 64 bit will be used more often in the future. I'm sure this is a .Net 2.0/WOW issue. If you're using a custom grid control, contact that company and find out what's wrong. Otherwise code your own or use something opensource.

Other than this DOM slowdown, 64 bit is perhaps a good choice for power users pushing the individual app RAM limit as you will go from 2GB per app with a 32 bit OS to about 4GB per app with a 64 bit OS from a practical standpoint, as you will probably only put 6GB or 8GB on any new machine. Though be forewarned, that with 64 bit machines/OSes, you may have trouble finding graphics or printer drivers. Also, almost no software in the trading world is compiled to 64 bit, specifically the real-time data vendors, eSignal, IB, etc.

Gains: We have experienced everything you stated about memory problems. While they aren't technically leaks, as the memory is released when Ninja closes, it is something that Ninja could handle better and thankfully they are aware of this.

BigD

[NinjaTrader_Dierk]

Thanks for this insight. We have Win 64bit support on our list of NT7 (end of this year) yet.

[Gains]

A solution... sort of.

Thanks BigD for sharing your Win64 experiences! Those are the results I had hoped for, aside from the DOM issues. I am going to have to play with that myself on another box.
For the mean time I have found a work around for my Vista core duo machine which is limited by hardware to only 2GB RAM and so probably would not benefit from a Win64 OS. I am still tempted to try it though just to see…

As I was searching the Net for information about memory issues with .NET and Vista I kept stumbling across the term "ReadyBoost". I had never heard of this Vista feature before.

http://www.microsoft.com/windows/products/windowsvista/features/details/readyboost.mspx

I immediately acquired a 4GB memory stick to try it out. I was very pleased with the results.
Whereas in the past I could bring Ninja and Vista to an almost total standstill by loading 30 to 32 charts, this is no longer the case. The RAM used by those ~32 charts (4 workspaces x 8 charts) could spike over 1GB. After adding the 4GB USB stick I started opening workspaces one after the other. RAM usage shot up rapidly to the 600 to 700 MB range but there it seemed to peak. I have successfully run 80 charts with this configuration and system RAM use has remained in the 700MB area.
The system is not totally stable but it is now very usable. The mysterious behaviors I still experience might be related to other issues like the workspace issue that Dierk is already dealing with in the next beta release.
So this is definitely worth trying if you have a Vista machine and are experiencing these RAM issues. I found a Verbatim 4GB stick labeled “ReadyBoost compatible” for just $40, so shop around.

Gains

[NinjaTrader_Dierk]

Thanks for sharing this info.