SMA indicator returning odd values...

[DrkShadow]

First, it must be noted that I'm working on an indicator and adding 30 stocks after the primary instrument.

DataSeries[] correlations = new DataSeries[31];
Initialize() {
    // There are 30 symbols
    for (i = 0; i < symbols.Length; i++) {
         Add(symbols[i], BarsPeriods[0].BasePeriodType, BarsPeriods[0].BasePeriodValue);
     }
    for (i = 1; i < Instruments.Length; i++) {
        // Correlations are kept historically for averaging
        correlations[i] = new DataSeries(this, MaximumBarsLookBack.TwoHundredFiftySix);

}

I'm using it in a 1-based manner. (30 stocks + 1 instrument = 31 total, so using less-than here is correct.) There are no array index exceptions, out of bounds, syntax errors, or anything else coming from the language itself.

As things go on, I calculate my own values. But we wish to smooth these values and not use the latest. To do this, I've been using the SMA indicator with my own data series:

OnBarUpdate() {
correlation = CMIFunctions.TradestationCorrelation(Closes[0], Close, length);
correlations[BarsInProgress].Set(correlation);

correlation = SMA(correlations[BarsInProgress], smooth)[0];
}

... seems simple enough, just average the last "smooth" values in the data series. Except correlation should be between -1 and 1. I was getting values from -2 to 5.5. That... can't be:

string corrList = "";
for (int i = 0; i < smooth + 1; i++) {
    corrList += "; " + Math.Round(correlations[BarsInProgress][i],3);
}
Print(BarsInProgress + ": Correlation: " + correlation + "; List" + corrList);
SMA testing = SMA(correlations[BarsInProgress], smooth);
                
string prevval = "Previous input values";
int j;

for (j = 0; j < smooth + 10 && j < CurrentBar; j++) {
    prevval += "; " + Math.Round(testing.Input[j],3);
}
prevval += "; Gave average values";
for (j = 0; j < smooth + 10 && j < CurrentBar; j++) {
    prevval += "; " + Math.Round(testing[j],3);
}
Print("Cached SMA" + prevval);

Yields output:

1: Bar: 441; Correlation List; 0.133; -0.133; -0.067
Cached SMAPrevious input values; 0.133; -0.133; -0.067; -0.067; 0.067; -0.267; 0.067; -0.067; -0.067; 0.133; 0.267; -0.067; Gave average values; 0.133; 0.167; 0.2; 0.267; 0.1; 0.267; 0.233; 0.233; 0.333; 0.4; 0.267; 0.433
1: Correlation Average: 0.133333333333333
2: Bar: 441; Correlation List; 0; 0; -0.133
Cached SMAPrevious input values; 0; 0; -0.133; 0.133; 0; -0.2; 0.067; 0; -0.067; 0.067; -0.067; 0; Gave average values; -0.067; -0.133; 0; -0.1; -0.233; -0.1; -0.167; -0.233; -0.167; -0.267; -0.2; -0.167
2: Correlation Average: -0.0666666666666665
3: Bar: 439; Correlation List; 0.067; -0.267; 0.067
Cached SMAPrevious input values; 0.067; -0.267; 0.067; 0.067; -0.2; -0.267; -0.067; -0.133; 0; 0.067; 0; 0.2; Gave average values; 0.033; 0.167; 0.2; 0.033; 0; 0.133; 0.133; 0.167; 0.2; 0.2; 0.3; 0.167
3: Correlation Average: 0.0333333333333334
4: Bar: 441; Correlation List; -0.067; -0.2; 0.067
Cached SMAPrevious input values; -0.067; -0.2; 0.067; 0.067; 0.2; -0.267; 0; -0.2; 0.2; 0; 0.067; 0.4; Gave average values; -0.033; 0.1; 0.133; 0.233; 0; 0.133; 0.033; 0.2; 0.1; 0.1; 0.267; -0.033
4: Correlation Average: -0.0333333333333335

<next bar>

1: Bar: 442; Correlation List; 0; -0.133; -0.067
Cached SMAPrevious input values; 0; -0.133; -0.067; -0.067; 0.067; -0.267; 0.067; -0.067; -0.067; 0.133; 0.267; -0.067; Gave average values; 0.133; 0.167; 0.2; 0.267; 0.1; 0.267; 0.233; 0.233; 0.333; 0.4; 0.267; 0.433
1: Correlation Average: 0.133333333333333
2: Bar: 442; Correlation List; -0.2; 0; -0.133
Cached SMAPrevious input values; -0.2; 0; -0.133; 0.133; 0; -0.2; 0.067; 0; -0.067; 0.067; -0.067; 0; Gave average values; -0.067; -0.133; 0; -0.1; -0.233; -0.1; -0.167; -0.233; -0.167; -0.267; -0.2; -0.167
2: Correlation Average: -0.0666666666666665
3: Bar: 440; Correlation List; 0; -0.267; 0.067
Cached SMAPrevious input values; 0; -0.267; 0.067; 0.067; -0.2; -0.267; -0.067; -0.133; 0; 0.067; 0; 0.2; Gave average values; 0.033; 0.167; 0.2; 0.033; 0; 0.133; 0.133; 0.167; 0.2; 0.2; 0.3; 0.167
3: Correlation Average: 0.0333333333333334
4: Bar: 442; Correlation List; 0.133; -0.2; 0.067
Cached SMAPrevious input values; 0.133; -0.2; 0.067; 0.067; 0.2; -0.267; 0; -0.2; 0.2; 0; 0.067; 0.4; Gave average values; -0.033; 0.1; 0.133; 0.233; 0; 0.133; 0.033; 0.2; 0.1; 0.1; 0.267; -0.033
4: Correlation Average: -0.0333333333333335

<next bar>

1: Bar: 443; Correlation List; 0.067; 0; -0.133
Cached SMAPrevious input values; 0.067; 0; -0.133; -0.067; -0.067; 0.067; -0.267; 0.067; -0.067; -0.067; 0.133; 0.267; Gave average values; 0.233; 0.133; 0.167; 0.2; 0.267; 0.1; 0.267; 0.233; 0.233; 0.333; 0.4; 0.267
1: Correlation Average: 0.233333333333333
2: Bar: 443; Correlation List; -0.133; -0.2; 0
Cached SMAPrevious input values; -0.133; -0.2; 0; -0.133; 0.133; 0; -0.2; 0.067; 0; -0.067; 0.067; -0.067; Gave average values; -0.133; -0.067; -0.133; 0; -0.1; -0.233; -0.1; -0.167; -0.233; -0.167; -0.267; -0.2
2: Correlation Average: -0.133333333333333
3: Bar: 441; Correlation List; -0.067; 0; -0.267
Cached SMAPrevious input values; -0.067; 0; -0.267; 0.067; 0.067; -0.2; -0.267; -0.067; -0.133; 0; 0.067; 0; Gave average values; 0.133; 0.033; 0.167; 0.2; 0.033; 0; 0.133; 0.133; 0.167; 0.2; 0.2; 0.3
3: Correlation Average: 0.133333333333333
4: Bar: 443; Correlation List; 0.067; 0.133; -0.2
Cached SMAPrevious input values; 0.067; 0.133; -0.2; 0.067; 0.067; 0.2; -0.267; 0; -0.2; 0.2; 0; 0.067; Gave average values; 0.1; -0.033; 0.1; 0.133; 0.233; 0; 0.133; 0.033; 0.2; 0.1; 0.1; 0.267
4: Correlation Average: 0.0999999999999998

(smooth = 2; variables have been renamed to protect the confidential. If something was renamed in error, refer to output provided.)

Etc. Obviously.. these values are incorrect. (For example bar 1 remember to use _two_ bars values, and the data should be: stock 1 should be 0 not 0.133.., stock 2 should be 0, not -0.066, stock 3 should be .0.1, not 0.033, and stock 4 should be -0.133, not -0.033.) I've looked through the SMA indicator code itself, and it seems straight-forward and obvious. I can't imagine what the issue is. Help?

Could this be a "missed bar" problem? If I don't set a value for one bar, and the next bar tell it to average over two bars... what does it use for that missed bar? How can I tell? Still, it should subtract that bar off the end when it recalculates the average.

This output is direct from the output window, no edits, no mixing.

Optimizations:
The SMA indicator _could_ take a couple optimizations... In if (CurrentBar >= Period), there's a usage of math.min() for the divisor in each case. Math.min() is unnecessary -- if currentBar >= Period, divide by Period; else, divide by CurrentBar + 1. The determination of double last could be done in the same way, if moved inside the if. The difference for last would be a couple extra bytes of code vs an extra if on every barUpdate. Anyway.. it just seemed like the SMA code was being pedantically efficient, so I thought I'd chip in ;-)

[NinjaTrader_Bertrand]

Welcome to our forums DrkShadow - thanks for the comments on the default SMA method, I've forward to development for review.

For your code issue - for debugging purposes I would recommend to simplify the script as much as possible, so for example to only add one series and then calculate your custom correlation value as you desired - this will present easier to review code / complexity while troubleshooting.

Next, are you getting expected values returned from your custom data series that you pass to the SMA method for smoothing?

How much data are you loading up for the instruments?

Please keep in mind with MultiSeries scripts, the OnBarUpdate() would only be called after all bars objects have satisfied their BarsRequired setting, so perhaps you're dealing with an Initialization issue here as the data normalizes for your calcs?

[DrkShadow]

I think this is concise enough, with nothing odd in the code, to demonstrate the problem. Three stocks, A, BAC, C, on top of primary instrument ES 06-12. The SMA is computed with length 2. The correlation values, over which the average is taken, are set one time only, and never changed. No one stock affects the value of any other.

I created a chart, ES 06-12, 3-minute bars going back 365 days. I added basics-sma to the chart as an indicator, and got output among the below.

I'd guess that some bars are getting values more than once. I'd then assume that the indicator's OnBarUpdate method is called for this second update (is that really how it works? when I .set() a value in a dataseries, indicators that are listening to that dataseries run the OnBarUpdate method? regardless of what index I set?), the SMA indicator substitutes off the bar on the far end, adds the new input value, and takes the average. When it's called the second time for that bar, it subtracts off an incorrect value, adds the newest end again, and that might be where the error occurs.

2012/03/02 12:04:00
2: Correlation List, -0.067, 0.467, 0.067
Previous input values, -0.067, 0.467, 0.067, 0.533, 0.067, 0.067, 0.067, -0.067, 0, 0.267, -0.267, 0.067
2: Correlation Average: 6.36666666666669
3: Correlation List, 0.133, 0.133, -0.067
Previous input values, 0.133, 0.133, -0.067, 0.267, 0, 0.267, 0.133, 0.133, -0.133, 0.133, 0.067, 0.133
3: Correlation Average: 106

#region Using declarations
using System;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.ComponentModel;
using System.Collections;
using System.Xml.Serialization;
using NinjaTrader.Data;
using NinjaTrader.Gui.Chart;
#endregion

// This namespace holds all indicators and is required. Do not change it.
namespace NinjaTrader.Indicator {
    /// <summary>
    /// </summary>
    public class basicssmatest : Indicator {
        #region Variables
        private string[] symbols = {
                "A",
                "BAC",
                "C"
            };

        private int length = 30;// How many bars do we use for the correlation calculation?
        private int smooth = 2;    // 
        
        // Store correlation values to average them
        private DataSeries[] correlations;

        #endregion

        /// <summary>
        /// This method is used to configure the indicator and is called once before any bar data is loaded.
        /// </summary>
        protected override void Initialize() {
            int i;

            for (i = 0; i < symbols.Length; i++) {
                Add(symbols[i], BarsPeriods[0].BasePeriodType, BarsPeriods[0].BasePeriodValue);
            }
            
            // Create correlation sets for each stock symbol
            correlations = new DataSeries[Instruments.Length];

            for (i = 1; i < Instruments.Length; i++) {
                // Correlations are kept historically for averaging
                if (smooth <= 256)
                    correlations[i] = new DataSeries(this, MaximumBarsLookBack.TwoHundredFiftySix);
                else
                    correlations[i] = new DataSeries(this, MaximumBarsLookBack.Infinite);
            }

            AutoScale = false;
            Overlay = true;

            // Set the barsRequired for each input length.
            BarsRequired = 40;    // say..
        }

        long calcBar = 0;

        // XXX Because NinjaTrader crashes (the indicator) if you ask if a DataSeries has a point/valid plot.
        bool barCalcd = false;
        DateTime bartime = DateTime.MinValue;

        protected override void OnBarUpdate() {

            // Because BarsRequired is for display, yet I want it for history.
            if (CurrentBars[BarsInProgress] < BarsRequired || CurrentBars[0] < BarsRequired)
                return;

            if (bartime != Time[0]) {
                Print(Time[0]);
                bartime = Time[0];
            }

            // Calculate each of the stock correlation values.
            if (BarsInProgress > 0) {

                double correlation;
                correlation = TradestationCorrelation(Closes[0], Close, length);
                correlations[BarsInProgress].Set(correlation);

                // This doesn't wait for all the required correlations to be available, so make sure you start it 
                // with at least smooth bars history (beyond what is required for calculation to start).
                string corrList = "";
                for (int i = 0; i < smooth + 1; i++) {
                    corrList += ", " + Math.Round(correlations[BarsInProgress][i], 3);
                }
                Print(BarsInProgress + ": Correlation List" + corrList);
                SMA testing = SMA(correlations[BarsInProgress], smooth);
                
                string prevval = "Previous input values";
                int j;

                for (j = 0; j < smooth + 10 && j < CurrentBar; j++) {
                    prevval += ", " + Math.Round(testing.Input[j], 3);
                }
                Print(prevval);

                correlation = SMA(correlations[BarsInProgress], smooth)[0];
                Print(BarsInProgress + ": Correlation Average: " + correlation);
                
            }
        }

        // Emulate Tradestation's version of Correlation function. Doesn't do a true statistical correlation.
        // While length is given, we actually use length + 1 bars. You must keep this in mind.
        // Notes: does not perform a true statistical correlation. Will not substitute.
        public static double TradestationCorrelation(IDataSeries one, IDataSeries two, int length) {
            int matches = 0;

            if (length < 1)
                return -2;

            for (int i = 0; i < length; i++) {
                // If they are both greater than their last bar, or both less, then it's a match.
                if (one[i] >= one[i + 1] && two[i] >= two[i + 1] || one[i] < one[i + 1] && two[i] < two[i + 1])
                    matches++;
            }

            // Because Tradestation scales the number of matches from -1 to 1: -1 is
            // complete chaos, 0 is half matching, and 1 is matching.
            return (2.0 * matches / length) - 1;
        }

        #region Properties
        #endregion
    }
}

[NinjaTrader_Bertrand]

Thanks DrkShadow, I agree on your analysis - you would need to filter your indicator OnBarUpdate() more specific for the various Barsobject / BarsInProgress calls it should listen to, currently the logic is run multiple times for each BarsInProgress call, which are in total 4 (ES as primary and the 3 added stocks) for your script. If you just need the data for calcs and the timing of the BarsInProgress events for the individual stocks can only work in BarsInProgress == 0 and filter out all other ones, the added instrument data could still be accessed via BarsArray[x] or Closes, Highs and Lows.

[DrkShadow]

The intention of the code was to calculate the bar's correlation on the bar's close for that stock symbol. The code does this by passing the correlation history for that single bar to SMA at the close of each bar.

function OnBarUpdate() {
// Calculate each of the stock correlation values.
if (BarsInProgress > 0) {

    double correlation;
    correlation = TradestationCorrelation(Closes[0], Close, length);
    correlations[BarsInProgress].Set(correlation);

    correlation = SMA(correlations[BarsInProgress], smooth)[0];
    Print(BarsInProgress + ": Correlation Average: " + correlation);
}
}

If I'm to wait for the last bar to close, so that the code only runs once instead of 3 times, how am I to detect that the last bar is the close? Does the primary instrument not close until all other instruments have been closed? Is there a property set somewhere that I can check? I was under the impression that the bars would close in any order, or even not at all, so I can't check to see if the barsinprogress is either 0 or instruments.length-1. How can I wait until the current call to OnBarsUpdate() is the _final_ call to OnBarsUpdate() for the current bar of the primary instrument?

[NinjaTrader_Bertrand]

DrkShadow, NT is a completely event based framework you're operating in, so you would know the current bar is for sure closed only if the first tick of the new (following) bar is seen -

http://www.ninjatrader.com/support/helpGuides/nt7/firsttickofbar.htm

Due to the event based nature, the closing tick of the current bar is the opening tick of the next, it's identical.

[DrkShadow]

Found the answer.

Note that "I created a chart, ES 06-12, 3-minute bars," and then added extra symbols like this:
Add(symbols[i], BarsPeriods[0].BasePeriodType, BarsPeriods[0].BasePeriodValue);

The particular error was using BarsPeriods in the Initialize() function. You can't. There is _no_way_ to obtain the period information from the primary instrument. The fix, then, is to specify the period type and period value as a parameter to the indicator. When they are set the same, SMA works as intended.

This code used the "default" period information: BasePeriodType was Minute (correct), but BasePeriodValue was 1 (should be 3). (If you actually try to use this information in the strategy optimizer, the code will error as BarsPeriods[0] is not defined.)

I still don't know why the added symbol updating more frequently than the primary symbol would cause a problem, but specifying the proper period value fixed the issue for me.

[NinjaTrader_Bertrand]

Glad to hear you could resolve your issue. Have a good weekend,