EMA output from custom DataSeries = wonky

[koganam]

Originally Posted by LiquidDrift

If this is true, than why do we not see it recurring in your output, only mine? Or were you seeing it in your output as well?

If what you're saying is true, then the EMA doesn't take into account new data entered into the DataSeries, which would make sense given the output.

It is there in the output of mine. The second instance shows a FP error too, but the first instance shows exactly as yours does.

Actually by trying to force the EMA value to integers (I just changed the EMA to use "4" instead of "5" and left the rest the same, I can confirm that, on my computer, at least, the second run is definitely due to FP errors. The problem is that I do not clearly see why they are occuring. And given that the problem seems non-existent if we force a recreation of the EMA class instance by first disposing of it, I must conclude that something really is, as you say, wonky.

Here is my output when I change the period to 4. I do expect the first term of the EMA to be exactly 260, and the other to be exactly 100, according to the universally accepted method of calculating an EMA (using 2 terms).

-------------------------
VALUE: 500
EMA: 260
VALUE: 100
EMA: 100
VALUE: 100
EMA: 100
VALUE: 100
EMA: 100
VALUE: 100
EMA: 100
-------------------------
VALUE: 500
EMA: 260.000000000008
VALUE: 100
EMA: 100.000000000013
VALUE: 100
EMA: 100.000000000022
VALUE: 100
EMA: 100.000000000036
VALUE: 100
EMA: 100.00000000006

[LiquidDrift]

But, hang on, in my second run, I get 322.2222222222 on the first EMA value, and then I get 233.333333333333 as the second value. We are definitely getting the same thing on the first run, but on the second we are not. And I don't get the FP fluctuations.

Maybe I should just make a complete Strategy and we can both run it exactly and see what we get.

[koganam]

Originally Posted by LiquidDrift

But, hang on, in my second run, I get 322.2222222222 on the first EMA value, and then I get 233.333333333333 as the second value. We are definitely getting the same thing on the first run, but on the second we are not. And I don't get the FP fluctuations.

Nor I. Believe me, I am really flummoxed on this one. I really do not want to bring out an ICE debugger to look at this one, but it is really making absolutely no sense so far. Even having to explicitly remove the EMA instance from memory with Dispose() in order to get the correct output makes no sense. (It will work, that is obvious, but we should not need to do that).

[sledge]

Originally Posted by LiquidDrift

But, hang on, in my second run, I get 322.2222222222 on the first EMA value, and then I get 233.333333333333 as the second value. We are definitely getting the same thing on the first run, but on the second we are not. And I don't get the FP fluctuations.

Maybe I should just make a complete Strategy and we can both run it exactly and see what we get.

I am curious and will run this tonight on my desktop and laptop

[sledge]

Originally Posted by sledge

I am curious and will run this tonight on my desktop and laptop

I see the problem. Do you??

[sledge]

Originally Posted by koganam

Nor I. Believe me, I am really flummoxed on this one. I really do not want to bring out an ICE debugger to look at this one, but it is really making absolutely no sense so far. Even having to explicitly remove the EMA instance from memory with Dispose() in order to get the correct output makes no sense. (It will work, that is obvious, but we should not need to do that).

I believe this testSeries.Set isn't working as you all expect.

It's acting like a FIFO queue... but you can't see it for some reason.. NT behind the doors is not letting it be known (or something else is going on).

Uncomment out the .Set for loop from 1..5... that makes the results even more whacky. Because we are stuffing the 500 on top, and trying to assign, but the assignment appears to only happen when there is nothing assigned... But you ARE NOT seeing that in your print statement... I'm a little too wasted to compute the EMA to be 100% sure..but that's my observation and I'm sticking to it.

Running the code below results in :

cnt=16
i=0 VALUE: 500 testSeries.Count=16
i=0 EMA:   166.666666666667
i=1 VALUE: 0 testSeries.Count=16
i=1 EMA:   0
i=2 VALUE: 0 testSeries.Count=16
i=2 EMA:   0
-------------------------
cnt=17
i=0 VALUE: 500 testSeries.Count=17
i=0 EMA:   277.777777777778
i=1 VALUE: 500 testSeries.Count=17
i=1 EMA:   166.666666666667
i=2 VALUE: 0 testSeries.Count=17
i=2 EMA:   0
i=3 VALUE: 0 testSeries.Count=17
-------------------------
cnt=18
i=0 VALUE: 500 testSeries.Count=18
i=0 EMA:   351.851851851852
i=1 VALUE: 500 testSeries.Count=18
i=1 EMA:   277.777777777778
i=2 VALUE: 500 testSeries.Count=18
i=2 EMA:   166.666666666667
i=3 VALUE: 0 testSeries.Count=18
i=3 EMA:   0
-------------------------
cnt=19
i=0 VALUE: 500 testSeries.Count=19
i=0 EMA:   401.234567901235
i=1 VALUE: 500 testSeries.Count=19
i=1 EMA:   351.851851851852
i=2 VALUE: 500 testSeries.Count=19
i=2 EMA:   277.777777777778
i=3 VALUE: 500 testSeries.Count=19
i=3 EMA:   166.666666666667
i=4 VALUE: 0 testSeries.Count=19
-------------------------
cnt=20
i=0 VALUE: 500 testSeries.Count=20
i=0 EMA:   434.156378600823
i=1 VALUE: 500 testSeries.Count=20
i=1 EMA:   401.234567901235
i=2 VALUE: 500 testSeries.Count=20
i=2 EMA:   351.851851851852
i=3 VALUE: 500 testSeries.Count=20
i=3 EMA:   277.777777777778
i=4 VALUE: 500 testSeries.Count=20
i=4 EMA:   166.666666666667
i=5 VALUE: 0 testSeries.Count=20

#region Using declarations
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Data;

using NinjaTrader.Gui.Chart;
#endregion

// This namespace holds all indicators and is required. Do not change it.
namespace NinjaTrader.Indicator
{
    /// <summary>
    /// Enter the description of your new custom indicator here
    /// </summary>
    [Description("Enter the description of your new custom indicator here")]
    public class wtf : Indicator
    {
        #region Variables
        // Wizard generated variables
            private int myInput0 = 1; // Default setting for MyInput0
        private EMA emaTest;
        public int j;
        private DataSeries  testSeries ;
        // User defined variables (add any user defined variables below)
        #endregion

        /// <summary>
        /// This method is used to configure the indicator and is called once before any bar data is loaded.
        /// </summary>
        protected override void Initialize()
        {
            Add(new Plot(Color.FromKnownColor(KnownColor.Orange), PlotStyle.Line, "Plot0"));
            Overlay                = false;
            j=0;
            this.testSeries        = new DataSeries (this,MaximumBarsLookBack.Infinite);
        }

        /// <summary>
        /// Called on each bar update event (incoming tick)
        /// </summary>
        protected override void OnBarUpdate()
        {
            // Use this method for calculating your indicator values. Assign a value to each
            // plot below by replacing 'Close[0]' with your own formula.
            if (CurrentBar < 15) return;
            Print( "-------------------------" );
            testSeries.Reset();
            
            int cnt = Math.Min( testSeries.Count, 256 );
            Print ("cnt=" + cnt );
            
            //if (cnt==16)
                testSeries.Set( 0, 500 );
/*
            for( int i=1; i<6; i++ )
            {
                testSeries.Set( i, 100.0 );
            }
*/            
            emaTest =  EMA( testSeries, 5 );

            
            if (j< 50)
            {
                for( int i=0; i<cnt; i++ )
                {
                    Print( "i=" + i + " VALUE: " + testSeries[i] + " testSeries.Count=" + testSeries.Count );
                    Print( "i=" + i + " EMA:   " + emaTest[i] );
                }
                j++;
            }
        
        return;
        }

        #region Properties
        [Browsable(false)]    // this line prevents the data series from being displayed in the indicator properties dialog, do not remove
        [XmlIgnore()]        // this line ensures that the indicator can be saved/recovered as part of a chart template, do not remove
        public DataSeries Plot0
        {
            get { return Values[0]; }
        }

        [Description("")]
        [GridCategory("Parameters")]
        public int MyInput0
        {
            get { return myInput0; }
            set { myInput0 = Math.Max(1, value); }
        }
        #endregion
    }
}

#region NinjaScript generated code. Neither change nor remove.
// This namespace holds all indicators and is required. Do not change it.
namespace NinjaTrader.Indicator
{
    public partial class Indicator : IndicatorBase
    {
        private wtf[] cachewtf = null;

        private static wtf checkwtf = new wtf();

        /// <summary>
        /// Enter the description of your new custom indicator here
        /// </summary>
        /// <returns></returns>
        public wtf wtf(int myInput0)
        {
            return wtf(Input, myInput0);
        }

        /// <summary>
        /// Enter the description of your new custom indicator here
        /// </summary>
        /// <returns></returns>
        public wtf wtf(Data.IDataSeries input, int myInput0)
        {
            if (cachewtf != null)
                for (int idx = 0; idx < cachewtf.Length; idx++)
                    if (cachewtf[idx].MyInput0 == myInput0 && cachewtf[idx].EqualsInput(input))
                        return cachewtf[idx];

            lock (checkwtf)
            {
                checkwtf.MyInput0 = myInput0;
                myInput0 = checkwtf.MyInput0;

                if (cachewtf != null)
                    for (int idx = 0; idx < cachewtf.Length; idx++)
                        if (cachewtf[idx].MyInput0 == myInput0 && cachewtf[idx].EqualsInput(input))
                            return cachewtf[idx];

                wtf indicator = new wtf();
                indicator.BarsRequired = BarsRequired;
                indicator.CalculateOnBarClose = CalculateOnBarClose;
#if NT7
                indicator.ForceMaximumBarsLookBack256 = ForceMaximumBarsLookBack256;
                indicator.MaximumBarsLookBack = MaximumBarsLookBack;
#endif
                indicator.Input = input;
                indicator.MyInput0 = myInput0;
                Indicators.Add(indicator);
                indicator.SetUp();

                wtf[] tmp = new wtf[cachewtf == null ? 1 : cachewtf.Length + 1];
                if (cachewtf != null)
                    cachewtf.CopyTo(tmp, 0);
                tmp[tmp.Length - 1] = indicator;
                cachewtf = tmp;
                return indicator;
            }
        }
    }
}

// This namespace holds all market analyzer column definitions and is required. Do not change it.
namespace NinjaTrader.MarketAnalyzer
{
    public partial class Column : ColumnBase
    {
        /// <summary>
        /// Enter the description of your new custom indicator here
        /// </summary>
        /// <returns></returns>
        [Gui.Design.WizardCondition("Indicator")]
        public Indicator.wtf wtf(int myInput0)
        {
            return _indicator.wtf(Input, myInput0);
        }

        /// <summary>
        /// Enter the description of your new custom indicator here
        /// </summary>
        /// <returns></returns>
        public Indicator.wtf wtf(Data.IDataSeries input, int myInput0)
        {
            return _indicator.wtf(input, myInput0);
        }
    }
}

// This namespace holds all strategies and is required. Do not change it.
namespace NinjaTrader.Strategy
{
    public partial class Strategy : StrategyBase
    {
        /// <summary>
        /// Enter the description of your new custom indicator here
        /// </summary>
        /// <returns></returns>
        [Gui.Design.WizardCondition("Indicator")]
        public Indicator.wtf wtf(int myInput0)
        {
            return _indicator.wtf(Input, myInput0);
        }

        /// <summary>
        /// Enter the description of your new custom indicator here
        /// </summary>
        /// <returns></returns>
        public Indicator.wtf wtf(Data.IDataSeries input, int myInput0)
        {
            if (InInitialize && input == null)
                throw new ArgumentException("You only can access an indicator with the default input/bar series from within the 'Initialize()' method");

            return _indicator.wtf(input, myInput0);
        }
    }
}
#endregion

[sledge]

Originally Posted by koganam

Where are you running this code? As a method, or in an event handler. Yes, it would make a difference.

I think I see what is going on now.

The EMA/SMA are calculated by rolling off the last EMA/SMA value and couple with every OnBarUpdate - this process keeps happening.

So even though you 'reset' your input data series, those last calculations for SMA/EMA are still there (and being pushed out on each OnBarUpdate), and the last value of the SMA/EMA period is subtracted out, and then your new value from data series [0] is rolled in.


Dispose() is going to be required to cause a fresh computation.


So definitely not a caching issue in this case.

I did see the caching in effect - (I've always heard about it, but seeing it was fun) - was using 2 SMA variables of the same period, and Dispose() of the 1st variable also disposed the 2nd variable. Change to the different periods, and as expected, they are separate again.


Modifying SMA into SMA2 and putting some prints there, made it all the more obvious.

[koganam]

Originally Posted by sledge

I think I see what is going on now.

The EMA/SMA are calculated by rolling off the last EMA/SMA value and couple with every OnBarUpdate - this process keeps happening.

Not quite. The statement is true for the SMA, that is being calculated on a rolling basis for efficiency. Not true for the EMA which is being calculated on 2 values. Of course that means that errors get propagated, as the new value includes the old, and so residuals are never lost.

So even though you 'reset' your input data series, those last calculations for SMA/EMA are still there (and being pushed out on each OnBarUpdate), and the last value of the SMA/EMA period is subtracted out, and then your new value from data series [0] is rolled in.


Dispose() is going to be required to cause a fresh computation.

Therein lies my confusion. Everytime a class is declared/assigned to, one expects a fresh copy of the object, with the old being completely discarded, and without having to first Dispose() of the old copy. Either that, or I must have gotten this whole OOP thing wrong. In fact, that is why the easiest way to initialize an ArrayList is to simply redeclare it: a trick we have been using for a long time, instead of using foreach to place initial values back into the ArrayList.

[sledge]

Originally Posted by koganam

Therein lies my confusion. Everytime a class is declared/assigned to, one expects a fresh copy of the object, with the old being completely discarded, and without having to first Dispose() of the old copy. Either that, or I must have gotten this whole OOP thing wrong. In fact, that is why the easiest way to initialize an ArrayList is to simply redeclare it: a trick we have been using for a long time, instead of using foreach to place initial values back into the ArrayList.

This does not compile.

smaTest = new SMA(testSeries,5);

'NinjaTrader.Indicator.SMA' does not contain a constructor that takes '2' arguments.

It looks like it does in the IndicatorBase code, which I am assume is where the constructor is.

This compiles, but doesn't give us much.

smaTest = new SMA();
smaTest.SMA (testSeries,5);

print ( smaTest[i] );

Error on calling 'OnBarUpdate' method for indicator 'wtf' on bar 15: Index was outside the bounds of the array.


This was really wild - no more index out of bounds..

smaTest = new SMA();
smaTest = smaTest.SMA (testSeries,5);

This causes runtime errors, with the long stack trace output window. I ignored and it kept processing something. Until I hit ABORT button, then NT totally disappeared!!!

[sledge]

Originally Posted by sledge

This was really wild - no more index out of bounds..

smaTest = new SMA();
smaTest = smaTest.SMA (testSeries,5);

This causes runtime errors, with the long stack trace output window. I ignored and it kept processing something. Until I hit ABORT button, then NT totally disappeared!!!

This actually worked, I had to hit IGNORE for the exception 50+ times (aborting would kick me out of NT),

but, survey says!, 380 is the SMA of the series.


i=0 VALUE: 500 Get=0 testSeries.Count=65 smaTest[0]=380
i=1 VALUE: 100 Get=0 testSeries.Count=65 smaTest[1]=300
i=2 VALUE: 200 Get=0 testSeries.Count=65 smaTest[2]=300
i=3 VALUE: 1000 Get=0 testSeries.Count=65 smaTest[3]=280
i=4 VALUE: 100 Get=0 testSeries.Count=65 smaTest[4]=100

[koganam]

Originally Posted by sledge

This does not compile.

smaTest = new SMA(testSeries,5);

'NinjaTrader.Indicator.SMA' does not contain a constructor that takes '2' arguments.

You should not use the new keyword.

[e-man]

Originally Posted by koganam

Therein lies my confusion. Everytime a class is declared/assigned to, one expects a fresh copy of the object, with the old being completely discarded, and without having to first Dispose() of the old copy. Either that, or I must have gotten this whole OOP thing wrong. In fact, that is why the easiest way to initialize an ArrayList is to simply redeclare it: a trick we have been using for a long time, instead of using foreach to place initial values back into the ArrayList.

would you have the same result if you changed the DataSeries to ArrayList or Double[]?

i recall reading somewhere that DataSeries was a special type of array that is always in sync (persistent) with the bars. if that's true, then the requirement to call Dispose in order to get a fresh object would make sense.

cheers,
-e