Possible Solution to Freezing of NT7 Charts

[aligator]

Dear Support:

NT Freezing Tip:

After many months of frustration with NT 7 freezing on loading small time frame charts (i.e. 1 - Min) and following all recommendations to no fixes, I finally noticed at least one possible reason for freezing and slow loading.

Most likely you are applying an indicator that is using a recursive process, such as a pattern recognition task, where the code is going back and fort looking for a certain pattern within the code limits over the entire data set. Assuming approximately 8000-9000 bars in 5 days of 1 min chart, a recursive calculation this many times is time consuming. Now, if you have more than one instance of this indicator, it is even worst.

The idea is to decide a range where that sought after pattern would be valid within that range; for example, the most recent 250 bars.

Solution:

NT7 provides a look-back method that can be used to limit the recursive calculation. Simply inserting the following statement under the "protected override void Initialize():"

ForceMaximumBarsLookBack256 = true;

makes life less frustrating.

Support: The NT explanation of forced look back is very brief and I may be using it incorrectly. Please comment if this is a valid work around and if values other than 256 for look back can be used.

Thanks.

[aligator]

Ok, I was too excited that freezing frustration is over. Well, it is not.

Although the above possible solution significantly reduces the freezing time, it will cause the indicator to miss many of the otherwise legitimate signals. Not acceptable.

I guess I did not understand how the force look back feature is applied. So, back to square one.

Cheers!

[sledge]

Can you modify what you are working on and still reproduce the issue, without including anything you want to keep secret?

[bltdavid]

Are you working with the 32-bit or 64-bit executable?

I've noticed that large amounts of historical data downloading (6 mos of ES tick data, for example) can cause the 32-bit NinjaTrader.exe to hang -- which makes sense, I've seen the NT7 32-bit process memory size in the Win7 Task Manager grow well into the 3GB+ range.

When I do the same downloads using the 64-bit executable, no such issues.

[Edit: And, yes, inside the db/tick folder I deleted the ES subdirs downloaded by the 32-bit version -- so that restarting NinjaTrader with the 64-bit version had to download all the ES historical tick data all over again -- so it should be a valid test.]

[bltdavid]

Anyways, anything to do with "freezing" and "memory" (where the word "recursive" is a proxy for the word "memory") ... and I immediately think: 64-bit version might solve that.

Please experiment/reproduce your problem with the 32-bit vs 64-bit NinjaTrader versions and let us know what you find.

[sledge]

The mention of "recursion" in this post causes me to shiver. I really hope aligator did not mean true recursion.

[bltdavid]

The mention of "recursion" in this post causes me to shiver.

Same here.

But recursion is an algorithm kind of thing, so the only way to know that recursion is being employed is by a relatively intimate amount of code knowledge (such as, if alligator wrote the code himself, or has a detailed explanation of the algo from the author ... without which, I wonder how the OP knows recursion was being used.)

[bltdavid]

Most likely you are applying an indicator ...

Can you identify the indicator you're referring to?

Is this a proprietary indicator?

Have you contacted the vendor regarding your experiences with their indicator?

[aligator]

Can you modify what you are working on and still reproduce the issue, without including anything you want to keep secret?

Thanks seldge;

Yes, nothing so special, just simple patterns code, and really not much to change as far as code efficiency. But, I found the offending indicator by elimination process and noticed the nature of recursive beast.

[aligator]

Are you working with the 32-bit or 64-bit executable?

I've noticed that large amounts of historical data downloading (6 mos of ES tick data, for example) can cause the 32-bit NinjaTrader.exe to hang -- which makes sense, I've seen the NT7 32-bit process memory size in the Win7 Task Manager grow well into the 3GB+ range.

When I do the same downloads using the 64-bit executable, no such issues.

[Edit: And, yes, inside the db/tick folder I deleted the ES subdirs downloaded by the 32-bit version -- so that restarting NinjaTrader with the 64-bit version had to download all the ES historical tick data all over again -- so it should be a valid test.]

Same issue on both 32 and 64-bit, we are just talking about 5 days of recent data on a top specs Win 10 system with plenty of memory. The funny thing is the freezing happens for 1-min data, and there are no issues with the same time span of tick or volume data.

Yes, deleting bin/db/data-Minute-tick-day-cashe folders and -Workspace file (and some times a fresh install) are routine in my shop.

[sledge]

Something wrong with your code then?

[aligator]

Anyways, anything to do with "freezing" and "memory" (where the word "recursive" is a proxy for the word "memory") ... and I immediately think: 64-bit version might solve that.

Please experiment/reproduce your problem with the 32-bit vs 64-bit NinjaTrader versions and let us know what you find.

No difference, 32 vs 64. Plenty of memory there, and no memory leak.

The way I think of recursive formulas is that any new data will cause a new calculation of the entire data set where the initial value is modified until all history bars are accounted for. Yes it is resource intensive but there are plenty of HRZs and Gigs available for an average chart. Same chart using equal ticks or volume will load fast with no issues.

[aligator]

Something wrong with your code then?

Quite possible.

[NinjaTrader_PatrickH]

Hello aligator,

Thank you for your post.

Using ForceMaximumBarsLookBack256 = true would hep to improve performance in an indicator that would run across the full set of available data on each bar. As you discovered, this also means that it does not look to the entire available data set if set to true. There are no other options available to set the number of bars to look back, it is either 256 or the full data set.

64 bit applications would have access to more system resources than 32 bit and you could see a performance improvement when switching from 32 bit to 64 bit. However, if the performance is not desired already on 64 bit you would need to look to other options.

We do have Performance Tips with items concerning NinjaScript at the following link: http://ninjatrader.com/support/helpG...ance_tips2.htm

However, I suspect the code running across the full data set on each bar is likely to be performance heavy. You may wish to reduce the Days To Load, Bars To Load, or Custom Range used in your Data Series menu of the chart or Strategies menu of the Strategies tab in the Control Center.

Please let me know if you have any questions.

[aligator]

Hello aligator,

Thank you for your post.

Using ForceMaximumBarsLookBack256 = true would hep to improve performance in an indicator that would run across the full set of available data on each bar. As you discovered, this also means that it does not look to the entire available data set if set to true. There are no other options available to set the number of bars to look back, it is either 256 or the full data set.

64 bit applications would have access to more system resources than 32 bit and you could see a performance improvement when switching from 32 bit to 64 bit. However, if the performance is not desired already on 64 bit you would need to look to other options.

We do have Performance Tips with items concerning NinjaScript at the following link: http://ninjatrader.com/support/helpG...ance_tips2.htm

However, I suspect the code running across the full data set on each bar is likely to be performance heavy. You may wish to reduce the Days To Load, Bars To Load, or Custom Range used in your Data Series menu of the chart or Strategies menu of the Strategies tab in the Control Center.

Please let me know if you have any questions.

Thank you PatrickH.

True, calculations look for patterns over the entire 3-days of data set on new bars and is performance intensive. However, what puzzles me is the performance on 1-m chart is significantly different for an equal tick and vol chart (1- m approximately = 2000 ticks or 6500 Vol for ES).

Thanks again.