Memory Leak?

[NinjaTrader_Josh]

I believe there is a memory leak issue when loading massive amounts of data. If not a memory leak, at least a memory usage issue. Normally NinjaTrader seems to take about 60-100mb of ram, but when I try to load a years worth of tick and volume data NinjaTrader's memory usage shoots up to 1.5gb+ and crashes shortly after.

When I loaded the years worth of tick data first NinjaTrader's memory usage shot up to 350mbs while it was loading and stayed at 350 after it finished. After the tick data was done I tried to load the years worth of volume bars and then the memory shot to 1gb and as it was downloading/saving it crept up to 1.5gb. At this moment you can probably imagine NinjaTrader being very sluggish with its various windows freezing and the impending crash.

This was experienced with a barebones workspace of NinjaTrader. Just the Control Center and one chart for Tick bars and one chart for Volume bars.

[NinjaTrader_Dierk]

This is not unexpected, since a year of tick data is a lot. This is not a memory leak but just the memory usage to load all the data. I would be quite surprised if it worked, though.

Quick math:
- 150.000 ticks per day on ES
- 200 trading days
- ~25 bytes per tick

-> 750 MB not counting additional overhead

[NinjaTrader_Josh]

Gotcha. Maybe it would be beneficial to NinjaTrader's performance/"lightweightedness" if it could read the data from the database on as it is needed instead of trying to store the whole thing in ram.

Just a little side note. It seems loading a year of tick data is manageable at ~400-600mbs of ram, but the killer is loading volume data. I tried to load volume for 30 days and just that busted 1.5gb of ram and crashed. 20 days of volume data loads fine with only ~300mb of ram though. NinjaTrader also doesn't seem to release the resources after I close the massive charts. After loading 20 days of volume the mem usage hovers at ~300mb until I restart Ninja.

[Gumphrie]

Agreed, I sometimes have similar problems if I have too many charts open and initiate a data feed connection.

Can't it be a bit more clever by junting stuff into disk space? I hate seeing those data popups that never close and don't tell you anything. Just end up having to restart Ninja.

[NinjaTrader_Dierk]

>> I tried to load volume for 30 days and just that busted 1.5gb of ram and crashed
We'll look into that.

>> I hate seeing those data popups that never close and don't tell you anything
This issue is not related. You'll see this popup as NT loads data from your provider. NT always loads data from your provider e.g. on the current day to not leave you with gaps.

[Gumphrie]

Originally Posted by NinjaTrader_Dierk

>> I hate seeing those data popups that never close and don't tell you anything
This issue is not related. You'll see this popup as NT loads data from your provider. NT always loads data from your provider e.g. on the current day to not leave you with gaps.

Yes, but they should go away and not stay around forever (with the abort button greyed out).

[NinjaTrader_Dierk]

- can you reproduce at will?
- what provider is that?

[Gumphrie]

I believe I have seen it on various data feeds, but the most consistent is AMP/Zen-Fire after you have many graphs open when you reconect. The 'Abort' button is always greyed out with them, so if the data doesn't come back there is no way to get rid of the popup except by restarting Ninja.

[NinjaTrader_Dierk]

You will see the Abort button greyed out on those feeds where an abort option is not supported. The button will be active on those feeds where abort would be supported.

[NinjaTrader_Dierk]

Originally Posted by uacvax

Gotcha. Maybe it would be beneficial to NinjaTrader's performance/"lightweightedness" if it could read the data from the database on as it is needed instead of trying to store the whole thing in ram.

Just a little side note. It seems loading a year of tick data is manageable at ~400-600mbs of ram, but the killer is loading volume data. I tried to load volume for 30 days and just that busted 1.5gb of ram and crashed. 20 days of volume data loads fine with only ~300mb of ram though. NinjaTrader also doesn't seem to release the resources after I close the massive charts. After loading 20 days of volume the mem usage hovers at ~300mb until I restart Ninja.

Thanks for providing the background info by PM. Here is the problem:
- volume bars even eat up more space than ticks bars, since they have not only one double value (open) but high/low/close too + 24 additional bytes
- going with volume 150 on a stock like AAPL even will "split" up e.g. a 1500 shares tick into 10 additional volume bars

-> the memory excess risk on 150 volume bars on stocks is much higher than just with tick bars

However: The next release will trap the exceptions you experienced and provide a meaningful error message.

Also: .NET 2.0 is a garbage collection system, meaning although NT properly release memory, you will not see an immediate decrease of memory footprint on task manager, since the .NET runtime could decide to keep this freed memory under its management. There is no way an application could control that.

[NinjaTrader_Josh]

Thanks for the explanations Dierk. I guess I'll just have to restrain myself from the urge to load astronomical amounts of data all at once.