Improved RSquared

[DaveE]

The current version of the NT RSquared indicator sets the value to 0 if the Variance over the period is 0. But the statistical "correlation coefficient" result in this case is actually "undefined" (not zero), so the value should not be Set.

Suppose I create a new corrected version of the indicator called RSquared2 that does not Set the value when Variance is 0.

I then want to do an SMA on RSquared2 for example

SMA(RSquared2(8),14)

But this will now output nonsense during periods when there is 0 Variance - since NT will put the Close value into the Input to SMA (which is unable to filter out bars that have not been Set). So the output of the SMA will be an "average" of closing prices and RSquared (not very useful)!

There are many other statistical functions that have an "undefined" result in certain circumstances (such as zero variance). NT seems to have made a token gesture towards modelling this by allowing the user to not Set all values. But the inability to detect whether or not individual Input values have been Set makes it impossible to build generic aggregation functions (such as a correct SMA) that can deal with non-Set bars.

Please I dont want a response saying "This is expected behavior".
I would like to see a commitment from NT to look into making the "Set" state of Input bars available to the user.

See Problem 3 here for more info.http://www.ninjatrader.com/support/f...ad.php?t=45066

[koganam]

The current version of the NT RSquared indicator sets the value to 0 if the Variance over the period is 0. But the statistical "correlation coefficient" result in this case is actually "undefined" (not zero), so the value should not be Set.

Suppose I create a new corrected version of the indicator called RSquared2 that does not Set the value when Variance is 0.

I then want to do an SMA on RSquared2 for example

SMA(RSquared2(8),14)

But this will now output nonsense during periods when there is 0 Variance - since NT will put the Close value into the Input to SMA (which is unable to filter out bars that have not been Set). So the output of the SMA will be an "average" of closing prices and RSquared (not very useful)!

There are many other statistical functions that have an "undefined" result in certain circumstances (such as zero variance). NT seems to have made a token gesture towards modelling this by allowing the user to not Set all values. But the inability to detect whether or not individual Input values have been Set makes it impossible to build generic aggregation functions (such as a correct SMA) that can deal with non-Set bars.

Please I dont want a response saying "This is expected behavior".
I would like to see a commitment from NT to look into making the "Set" state of Input bars available to the user.

See Problem 3 here for more info.http://www.ninjatrader.com/support/f...ad.php?t=45066

NT does allow you to check for the validity of the value in a DataSeries. i.e., even though Reset() is incorrectly documented to say that the value is set to zero, when it in fact seems to be set to the close, you can still determine if the value is actually valid.

It is the DataSeries.ContainsValue(intbarsAgo) method.

ref: http://www.ninjatrader.com/support/h...ries_class.htm

[lefevreed]

This doesn't seem to be an NT problem, but a logic problem. How can you calculate the moving average of a null (not set) value in the first place? If you use null instead of 0, any bar in the look-back period that was set to null would create an error. You certainly could use custom coding as a workaround for the moving average function, but mathematically you can't calculate an average of a null value, by definition the value doesn't exist.

[koganam]

This doesn't seem to be an NT problem, but a logic problem. How can you calculate the moving average of a null (not set) value in the first place? If you use null instead of 0, any bar in the look-back period that was set to null would create an error. You certainly could use custom coding as a workaround for the moving average function, but mathematically you can't calculate an average of a null value, by definition the value doesn't exist.

One would certainly need to write a custom coded SMA to do that. It should be pretty simple to test and exclude any values where ContainsValue() is false. IOW, I agree with you: a null value is not zero; it simply does not exist, and should be excluded.

[lefevreed]

Yes Koganam, I was going to clarify, but you beat me to it!

Basically you have to figure a way to deal with null references. You either assign them a value, like say "0", or you exclude them as Koganam suggested. However, by excluding them, your 14 period moving average will be based on the last 14 non-null bars. Which can be anywhere from 14 to an undetermined number of bars in the past.

[koganam]

Yes Koganam, I was going to clarify, but you beat me to it!

Basically you have to figure a way to deal with null references. You either assign them a value, like say "0", or you exclude them as Koganam suggested. However, by excluding them, your 14 period moving average will be based on the last 14 non-null bars. Which can be anywhere from 14 to an undetermined number of bars in the past.

That is one perennial truth of statistical measurement. One has to decide what is significant, then deal with the consequences of that decision. In this case, how does one decide?

[DaveE]

However, by excluding them, your 14 period moving average will be based on the last 14 non-null bars. Which can be anywhere from 14 to an undetermined number of bars in the past.

I am not suggesting that would be a useful way to proceed.
Instead you need to look at what the indicator is supposed to achieve. Generally when Period is an input parameter this should mean "I want the most accurate value for the function looking only at the last Period bars, and I only want to start counting when I get the first 'non null' input value".
So if the function is R Squared with period of 8 bars, then
a) It does not make any sense to set a value before we have got the first 8 bars (starting counting at the first non-null value). This means that what NT calls the "unstable period" should have all values set null.
b) If there is a "null" (unset) value in the last 8 bars then it should be ignored (as per the SQL standard for aggregate functions).
See also the Microsoft standard for null handling in Linq queries http://msdn.microsoft.com/en-us/library/bb738551.aspx

The above policy cant work with the current state of existing NT built-in indicators since (for one thing) they dont wait until they have sufficient bars before using Set.

Going back to my original example: Lets say there were corrected indicators that worked according to the above policy:
SMA2(RSquared2(8),14)
would work like this:
RSquared2(8) would normally start using Set at bar 8. But lets say the first 8 bars of its input happen to have 0 Variance, then the function is undefined so it would also leave bar 8 un-Set.
Lets say Variance for last 8 bars only becomes non-zero at bar 10, then RSquared2(8) will only start using Set at bar 10. This illustrates that what NT are calling the "unstable period" is dependent on a combination of the user defined Period and the actual input data.
Now the output of RSquared2 becomes the Input for SMA2(14):
This will using

if (Input.IsValidPlot(CurrentBar))

(facility added in NT 7.0.1000.7) to wait until it finds a bar that has been Set in the Input. In this example at bar 10 of the Input SMA2 knows that Rsquared2 has completed its initialization, and is ready to be used. SMA2 now waits another 14-1 bars before it has enough bars to start setting its own value. So at the 23rd bar it looks at the last 14 bars and does the following:
Sum= sum of non-null values in last 14 bars. N=count of non-null values in last 14 bars. If N>0 then Set Sum/N, else output is null.

[DaveE]

But the inability to detect whether or not individual Input values have been Set makes it impossible to build generic aggregation functions (such as a correct SMA) that can deal with non-Set bars.

I have just found out (thanks to Brett for pointing out the new documentation) that it is now possible to detect whether individual Input values have been set.
This is done using

if (Input.IsValidPlot(CurrentBar))

(a facility added in NT 7.0.1000.7)
Just so long as you remember that the indexing for IsValidPlot is the opposite to the indexing for ContainsValue, then its now possible to write generic aggregation functions (such as a correct SMA) that can deal with non-Set bars.