Large memory use

[ETFVoyageur]

Spurred by some of Mr. K's suggestions, I created a linear regression function of my own to use instead of LinRegSlope() from my custom indicator. It:

• Is now MUCH faster
• No longer has the multi-gigabyte memory bloat

What I think was happening was that NinjaTrader caches things to improve performance. Unfortunately, that means the last reference to them never goes away, and so the temporary objects NEVER get garbage collected. (I even put in explicit calls to GC.Collect() and all they did was slow things down -- no improvement in memory footprint.)

That may be a sane performance tradeoff for a chart or a fairly small Market Analyzer, but is is a real loser for big Market Analyzers, such as mine (one 900 rows, the other 1600 rows, both about 10 columns -- most of which were my custom indicator with various periods). My custom indicator is not a problem in that scenario, but having it call LinRegSlope() was blowing size way up. Even a vastly simplified test case used more than 10GB, and took quite awhile to do so.

Anyway, now that I just call my own function instead of LinRegSlope() performance is quite acceptable for these Market Analyzers..

--EV

[koganam]

Spurred by some of Mr. K's suggestions, I created a linear regression function of my own to use instead of LinRegSlope() from my custom indicator. It:

• Is now MUCH faster
• No longer has the multi-gigabyte memory bloat

What I think was happening was that NinjaTrader caches things to improve performance. Unfortunately, that means the last reference to them never goes away, and so the temporary objects NEVER get garbage collected. (I even put in explicit calls to GC.Collect() and all they did was slow things down -- no improvement in memory footprint.)

That may be a sane performance tradeoff for a chart or a fairly small Market Analyzer, but is is a real loser for big Market Analyzers, such as mine (one 900 rows, the other 1600 rows, both about 10 columns -- most of which were my custom indicator with various periods). My custom indicator is not a problem in that scenario, but having it call LinRegSlope() was blowing size way up. Even a vastly simplified test case used more than 10GB, and took quite awhile to do so.

Anyway, now that I just call my own function instead of LinRegSlope() performance is quite acceptable for these Market Analyzers..

--EV

Aha! So it might have been just as I suspected. Whenever I see a loop, coupled with a performance issue, I get leery. I am glad that you rewrote the LinRegSlope to use a more efficient algorithm. As the formula for determining the coefficients of a linear regression can be easily reduced to specific summations, a for loop is just a poor choice of algorithm, and your experience shows exactly why. For loops become onerous once the working set gets large.

However, a troll through the magic wrappers shows the caching scheme that you talk about, and is why using a named instance of the class avoids the overhead of the cache search.

[ETFVoyageur]

Actually their algorithm looks OK -- they do use series formulae to compute sum(X) and sum(X^2). Their loop is to compute sum(X*Y), and I presume the call to SUM() used to compute sum(Y) has a loop inside that. I do not know how one would avoid those loops.

My concern is with their caching in the Magic Code. I am not sure just how it works, but I am worried about two things:
1) They search the cache with a linear search (actually two of them). That is fine for small caches, but may not scale well.
2) The cache would hold a reference to an object, preventing the object from getting garbage collected. Fine for small cases, such as charts and small Market Analyzers, but I'm not sure it scales well to big Market Analyzers. I'm suspicious that is how my simple test case grew to over 10GB.

--EV

[koganam]

Actually their algorithm looks OK -- they do use series formulae to compute sum(X) and sum(X^2). Their loop is to compute sum(X*Y), and I presume the call to SUM() used to compute sum(Y) has a loop inside that. I do not know how one would avoid those loops.

My concern is with their caching in the Magic Code. I am not sure just how it works, but I am worried about two things:
1) They search the cache with a linear search (actually two of them). That is fine for small caches, but may not scale well.
2) The cache would hold a reference to an object, preventing the object from getting garbage collected. Fine for small cases, such as charts and small Market Analyzers, but I'm not sure it scales well to big Market Analyzers. I'm suspicious that is how my simple test case grew to over 10GB.

--EV

Yes, the caching works well enough for small cases, but this same performance degradation issue, once when I had an involved chain of calls, with COBC = false, is what clued me into now using named instances of all classes that I call. That completely bypasses the cache searches.

And no, a loop is always a performance hit. How onerous it is depends on just how often the loop is called. Once can ALWAYS do a summation by using a difference in accumulation.

[ETFVoyageur]

I'll have to revisit that .. you have a good point.

Their sum(X) and sum(X^2) formulae would probably be faster done as differences., even though there is no loop involved. You would be replacing multiply / divide operations with add/subtract operations.

I agree that sum(Y) could clearly be done as a difference operation instead of a loop.

sum(XY) cannot be done as a difference, as they have it today -- the x-value that corresponds to each y-value changes each time. It seems to me that if one used the bar# instead of the bar offset for x, though, that then you could use a difference.

Not #1 on my priority list just now, but worth looking at before I am done.

BTW: I wonder about indicator efficiency in general, and I wonder how much it matters .. perhaps not that much. I noticed these lines in the SMA indicator

                if (CurrentBar >= Period)
                    Value.Set((last + Input[0] - Input[Period]) / Math.Min(CurrentBar, Period));

The Math.Min check is not needed, since the statement is guarded by an i"if" that has already done the check.

--EV