NT7 memory consumption

[NinjaTrader_Dierk]

Guys,

As you probably know we applied several performance and memory improvements to the NT7 code base. Our internal tests are more than promising. I would not want to provide actual numbers at that time, but the improvements are significant so far.

However, internal testing always will be limited. We wanted to run some reality checks by verifying against your "real world" scenarios (we are targeting around 10 different scenarios).

Could you please mail me to dierk [dot] droth [at] ninjatrader [dot] com your "typical" scenario where you experience memory issues with NT6.5?
Your mail should include:
- a reference to this thread
- a zip archive holding these folders:
* <my documents>/NinjaTrader6.5/workspaces
* <my documents>/NinjaTrader6.5/templates
- step by step instructions on how "to hit the wall"
It's crucial that you would provide the folder/files mentioned above in your mail.

Important: Please make sure your setup
- only would use NT standard indicators
- only would use NT standard strategies or simple as possible custom strategies (please include those as well)
- your setup rather would take minutes than hours to experience the issue in question
- would run on the main monitor and not require additional screens
This is to ensure we really focus on the subject.

I'm unsure on how much feedback we'll get. Please don't be disappointed if I would reject your input in case we would get way more than the targeted 10 setups.

Thanks in advance for your continued support
----------------------------------------------------------------------------------------------------------------------------------------
Results:
Setup1, many tick/volume/second based charts on high volume futures, NT startup and chart recovery:
NT6.5: blew up after >5min with OutOfMemory exception and 1.2GB RAM peak usage
NT7: finished after 75 secs, RAM usage always was below 100MB

Setup2, backtesting of a multi-series strategy on a basket of instruments (I created a similar setup, since user could not disclose all details):
NT6.5: RAM usage >1.3GB, blew up with OutOfMemory exception after >6mins
NT7: RAM usage < 430MB, done after 80 secs

Setup3, backtest of SampleMaCrossOver on 50 ticks ES series, 2009-03-15 through 2009-06-08:
NT6.5: RAM peak usage 1.2GB, 550MB after backtest complete, took > 4mins
NT7: RAM usage always < 110MB, backtest completed after 40secs

Setup4, export 1 year of ES continuous contract tick data
NT6.5: User reported was not possible and only could be done in chunks of 1-3 months
NT7: RAM usage < 500MB, took ~5mins

Setup5, backtest of SampleMaCrossOver on 50 ticks ES series, 2008-06-01 through 2009-06-09:
NT6.5: User reported was not possible due to OutOfMemory exception
NT7: RAM usage < 140MB, took < 3mins

[designer]

When is NJ7 being released now?

[NinjaTrader_Kyle]

Hello designer,

Thank you for your post.

NinjaTrader 7 is scheduled to release in beta by the end of the month.

[ctrlbrk]

Dierk,

As you already know from our previous exchanges, I have hit this limitation very often. I have not "applied" to help by providing my strategies because 1) proprietary DLL's are involved, and 2) well, they are "my" strategies and I don't trust anyone and 3) [most importantly] the largeaddress patch has made them far less frequent and annoying.

But, if you don't get the desired response and really need some help, even though it would be some work for me to re-program some of the strategies, I will do it in the interest of helping the community (and myself) improve NT7. Let me know if it is required.

BTW, I applaud your post asking for help from the community. I hope to see more posts like this in the future, it would signal a change for the better, in my opinion.

Mike

[NinjaTrader_Dierk]

@ctrlbrk, thanks for your feedback. I will come back to you in case.

[NinjaTrader_Dierk]

Added results to original post

[whitmark]

Dierk,

That is promising news. Are you looking for chart configuration examples or strategy analyzer examples where the data load can be extensive? In the later case, are templates and workspaces needed?

[Gains]

Dierk,
Wow! This is fantastic news, thanks for sharing it.
Well done, I can hardly wait to take my own test drive.
Gains

[NinjaTrader_Dierk]

Dierk,

That is promising news. Are you looking for chart configuration examples or strategy analyzer examples where the data load can be extensive? In the later case, are templates and workspaces needed?

I'm looking for both. Templates/workspaces would not be needed for strategy analyzer related issues.

In fact market analyzer setups would be interesting as well.

[NinjaTrader_Dierk]

Add new results to original post

[ChiTrader2000]

NinjaTrader_Dierk and NinjaTrader development team,

This is great news!

Thank you very much for dealing with this issue!

ChiTrader2000

[whitmark]

Dierk, impressive results on Setup 3! For extra credit, would you consider loading up the current contract with a year of ES tick data from previous contracts and rerunning the test? The implication from your initial results on memory consumption is that simple strategies on single timeframes with minimal plots/dataseries may realize a tenfold+ improvement in backtesting periods. Someone pinch me

Thanks,

Whitmark

[NinjaTrader_Dierk]

I will try if you could provide an import file (e.g. on some download area, you could try Help->Mail to support+attachment). I would not want to spend time right now to merge stuff together...

Also: the "ratio" on RAM efficiency gained certainly will vary dependent on actual setup. So please take it with a grain of salt.

[ctrlbrk]

Setup3, backtest of SampleMaCrossOver on 50 ticks series, 2009-03-15 through 2009-06-08:

NT6.5: RAM peak usage 1.2GB, 550MB after backtest complete, took > 4mins

NT7: RAM always usage < 110MB, backtest completed after 40secs

Happy Dance!



Mike

[whitmark]

I will try if you could provide an import file (e.g. on some download area, you could try Help->Mail to support+attachment). I would not want to spend time right now to merge stuff together...

Also: the "ratio" on RAM efficiency gained certainly will vary dependent on actual setup. So please take it with a grain of salt.

Are there any eSignal users out there who have been keeping up their continuous ES ##.## symbol or others who maintain exported ES continuous contract data at the tick level for this stress test of NT7 memory consumption? If not, when I have more time, I'll have to splice what I can from backfill data from each separate contract. No need to backadjust for this excersize.

Thanks,

Whitmark