NT8 Memory Optimization Issues

[dvance]

Hi!

I have written a Strategy in NT8, but am experience stability issues due to excessive memory use. In order to get around that, I'd like some help understanding the architecture NT8 uses when it comes to data streams, as well as somebody perhaps pointing out to me what's the recommended approach to handle a large number of data series with medium amounts of historical data.

First of all, is there an undocumented option (or a workaround) to set MaximumBarsLookBack to something other than 256 or Infinite? My system requires about 2000 bars lookback to do some calculations, so don't really need infinite, but 256 does not work either.

My trading system uses about 60 distinct data streams (stocks added via AddDataSeries in my code), if I run a few instances of the system on different stocks (but the same stock data streams are used internally between all), I end up with 70-80 distinct data streams in total. My memory consumption goes to over 11 GB in just a few days. It usually grows really high on the weekend when there is no incoming data -- could this be an NT8 bug?

11 GB sounds like too much as even several years worth of data for all the stocks should be under 2 GB.

Does NT8 automatically recognize that different strategies are requesting the same data and instead of instantiating multiple copies it uses a single one?

Is there a way to force NT8 to truncate the data series to a value (other than 256)?

Is this expected behavior or is there something that I've missed out in my own code that's resulting in this massive memory leak?

Any suggestions and thoughts welcome!

Thanks!

[NinjaTrader_ChelseaB]

Hello dvance,

Thank you for your note.

Does the strategy create new objects such as drawing objects on each new bar or in OnMarketData?

Is this occurring when running in real-time?

If you do not need to call more than 256 bars in the past, leave MaximumBarsLookBack set to TwoHundredFiftySix for efficiency. There are no other settings for this.

Below is a public link to the help guide.
https://ninjatrader.com/support/help...rslookback.htm


There is an instrument thread that does subscriptions for real-time data, NinjaScript will be consumers that thread will feed.

Objects being created may be causing the memory growth.

[dvance]

Hi Chelsea,

The strategy does not create new drawing objects at all.

Yes the problem occurs when running in real time.

I do need to call more than 256 bars in the past.

Could you please walk me through how 60 real-time data streams could use up 11+ GB of RAM in less than 48 hours (especially on the weekend when the stock exchange is closed)? Is that something you've tested on your platform -- in case it is and it has behaved differently, then I should spend more time looking through my own code to make sure it's not some other issue. Having said that my code is rather simple in terms of variable generation etc. so the likelihood of a memory leak this large I believe is small.

[NinjaTrader_ChelseaB]

Hello dvance,

60 added instruments means OnBarUpdate triggers for every instrument.
Logic must be run each time OnBarUpdate (or other data driven methods) runs.
New objects being created (such as ints, doubles, strings, orders, drawing objects, datetime objects, etc) in OnBarUpdate means those objects will be added to memory.

A script with no logic at all in OnBarUpdate is unlikely to grow to 11GB in memory.

I'm happy to create a test script that adds 60 instruments and has no logic in OnBarUpdate or any other methods. I'll let you know the results.

[NinjaTrader_ChelseaB]

Hello dvance,

I've created a script that adds 60 instruments, has no complex logic, does not create new objects, and prints the instrument name on each new session (just to show the script is still working and triggering OnBarUpdate()). No other workspaces, charts, or scripts were running during this test. The connection was Kinetick which did not experience any disconnects.

I've allowed this to run from Thursday through the weekend. The memory did not grow in size.

This lets us know that adding 60 instruments to a script in of itself does not cause a memory leak.

Attached are screenshots of the test when it was enabled, one taken after hours during the weekend, and one after the session start today.

[NinjaTrader_ChelseaB]

Hello dvanc,

Attached is the script I have tested if you also want to test this with all workspaces closed, all other charts closed, and no other indicators or scripts running to see if your memory grows.