DataSeries White Paper

[bltdavid]

Regarding Patrick's code: MyDataSeries[0] = Close[0]
Can I continue to add to MyDataSeries?
Ex: MyDataSeries[0] = Open[0] without overwriting the initial Close[0]?

No, your second assignment will override the value stored by the first assignment.

[Edit: Unless each assignment is happening in a separate invocation of OnBarUpdate, because then the most recent bar accessed by MyDataSeries[0] is actually a new bar, because OnBarUpdate is called when that new bar closes, so the indexer '[0]' is automatically updated under the hood (for free) to always reference the most recently closed bar.]

[Edit-2: Spoiler alert: The above is a convoluted way of saying "slot". Keep reading, "slot" will be introduced soon, and then come back here later and read again after you've achieved a good feel for "slot".]

Would this code be the equivalent:

double todaysClose = Close[0]; ????

Yes, they are equivalent -- but isn't it obviously so? I mean, once you assign a value to a variable, then naturally the variable on the left of the equals sign is equivalent to the value on the right of the equals sign.

Why would I choose a DataSeries object over a simple typed variable?

You would only choose an xxxSeries object over a simple typed object if you have values that need to be (re)calculated and remembered for every bar.

But, also, because you get some additional memory management for free. I mean, each and every DataSeries (or any instance of an 'xxxSeries' class) has exactly one value maintained for each bar on your chart. These values are maintained in parallel, such that literally any xxxSeries object accessed with suffix [0] always means the value associated with the current bar, any xxxSeries object accessed with suffix [1] always means the value 1 bar ago, and so on.

[Edit: When NinjaTrader wrote their xxxSeries classes, they effectively have to shift all values in the "array" one position to the left, to make the indexer '[0]' free for the new value from the new bar. That's what I mean by NT's DataSeries memory management: NT makes room for the new value at '[0]' for all xxxSeries objects attached to the chart just before calling OnBarUpdate -- it does this automatically for us.]

[Edit-2: I'm almost positive that no shifting of data actually takes place. I'll cut off my left nut if I'm wrong about this: NT engineers used "indexed properties" for the '[0]' indexer, there is no "shifting" to free up indexer '[0]'. See my additional answers below.]

But the real beauty is a bit more complex, and under the hood. The engs at NinjaTrader made each and every indicator a DataSeries. Think about an EMA, or Stochastics. They all have values that are calculated and stay associated with each bar, right?

Consider that EMA(10)[0] is accessing the value of the EMA(10) for the current bar, and that EMA(10)[10] is accessing the value of EMA(10) exactly 10 bars ago. The suffix of '[0]' and '[10]' are indexers, and this is DataSeries syntax applied to an indicator.

DataSeries' look like an array to me except the "[ ]" are used to designate time instead of a piece of memory... Make sense?

Exactly. They do look like an array, the "[ ]" is known as the indexer, and you are exactly right to see that the indexer specifies an age based value (not time, think age). NinjaTrader calls this age based index the BarsAgo index, which makes perfect sense.

[Edit: Why age? Because it is a better metaphor. Open[3] is the bar 15 minutes ago on 5 minute bars, or 3 days ago for daily bars, but what if each bar is a 500-Tick bar, or a 1000-Volume bar, or a Renko bar? Time as a metaphor breaks down, but age continues to be completely correct in its meaning -- here, age literally means n bars ago.]

What I'm hoping to do (and could with traditional C# patterns) is create a custom object that represent various objects. (OOP 101)
Ex:
class TradingDay {
public double Open { get; set; }
public double High { get; set; }
public double Low { get; set; }
public double Close { get; set; }
public double Volume { get; set; }
public double HotIndicatorVal { get; set; }
List<double> ...
}

Sure, you could do this. But you'd be re-inventing so much of the wheel that has been provided to you that, well, it would be nuts. I say that kindly, and with respect.

I mean, consider that NinjaTrader engs could have designed a single class structure to contain all data for each bar inside it, and then made a DataSeries of these objects, such that each instance of TradingDay stored in the DataSeries would represent a separate bar ... sorta like where your code is heading above.

But this design becomes cumbersome later. Why? Because some new values come along that are also associated with each bar -- and logically they should be stored in this class, right?

NT engs decided that, from the point of view of extensibility, multiple parallel arrays containing just one piece of data were better than a single array of a monolith structure that contained all the data. Why? Because it's easier to manage, and much easier to extend -- so they focused on making sure that Open[0] is treated specially to mean the open price of the most recently closed bar, and EMA(24)[0] means the value of EMA(24) of the most recently closed bar, and so on. The 'parallelness' accessed via the indexer '[0]' is maintained by NinjaTrader, for free.

Let's say you write a MillionDollar indicator that takes 3 arguments, x, y, z, so it would be called like MillionDollar(x, y, z) -- but because every indicator is also a DataSeries, we can also use an indexer to say MillionDollar(x, y, z)[n] to access the MillionDollar(x, y, z) value n bars ago.

Consider that the success of third party product called Bloodhound is predicated (partially) on this DataSeries data structure -- the multiple parallel arrays of simple values VS the single array of large class of values -- the engineers at NinjaTrader made some wise choices.

[bltdavid]

I'm trying to dig deeper into the DataSeries feature... What is it? Is a some sort of mutable object that I can stuff anything into? It seems like a powerful container, but the documentation on it:

https://ninjatrader.com/support/helpGuides/nt7/?dataseries_class.htm

is lacking in description, details and usage.

I see a "Set()" method... Is this just a wrapper around an array, Custom Object?

DataSeries is a class, which can be thought of as encapsulating an array.

Yes, thinking like it's a "custom array" is a very good metaphor.

But don't say object, all xxxSeries are classes, not objects.

xxxSeries use the power of the C# feature known as "indexed properties" to achieve the illusion that the indexer '[0]' is the youngest value added to the array (because in a regular array, the indexer [0] accesses the oldest element in the array.)

And, yes, the Set method is just a wrapper.

Effectively, the engineers at NinjaTrader have written something like,

public class DataSeries
{
    private List<double> DataList = null;

    public DataSeries()
    {
        DataList = new List<double>();
    }

    public double this[int indx]
    {
        get { return DataList[DataList.Count - indx - 1]; }
        set { DataList[DataList.Count - indx - 1] = value; }
    }

    public void Set(double NewValue)
    {
        DataList.Add(NewValue);
    }
}

See that? When 'indx=0' then 'DataList.Count - indx - 1' is the last element added to the array. Study this code carefully.

The goal is: if Arr is a DataSeries, accessing Arr[0] actually accesses the last element of the internal array DataList, aka, the most recent value added to the array -- such is the power of the "indexed property" in C# to create this illusion.

https://msdn.microsoft.com/en-us/library/aa288464%28v=vs.71%29.aspx

[RobVig]

ADDENDUM: I didn't see your last post (#17) before I wrote this...


David,

I gathered that a DataSeries is "like" an inverted array. What I still fail to understand is the mutability of a single "bar" of a DataSeries. Perhaps you can answer this:

1. Arrays are of a specific type Ex: int, double, string... Does this mean I can only store a single type in a (how would I term this in the Ninja world?) single bar instance of a DataSeries?

Think of a custom enumerable object / List<T> in C#, or a DataSet in R versus a simple array... Where does an "NT" DataSeries fit in?

If I write DataSeries.Set(High - Low) Can I then add a second item (Open) and subsequently a third (Close) to that same instance / bar location, or would that overwrite "High - Low"... I'm getting that you can only have a single item Ex: DataSeries.Set(x) and not DataSeries.Set(x,x) then add DataSeries.Set(x,x,x)...

I'm trying to find out what the data structures are in NT and how far I can push them.

Thank You,
Rob

[RobVig]

Now you're talking!!! (Post #17)

So ultimately is it true to say that:

1. DataSeries is of a single type
2. DataSeries can hold multiple distinct values
3. DataSeries is mutable

[bltdavid]

1. DataSeries is of a single type

Correct.

Values stored inside a DataSeries are all of the same type, and that type is "double".
Values stored inside a IntSeries are all of the same type, and that type is "int".
Values stored inside a BoolSeries are all of the same type, and that type is "bool".

And so on ...

To say a DataSeries of ints is a bit of a misnomer. If you need a DataSeries of ints, then just use an IntSeries or LongSeries.

But nothing is stopping you from storing whole number values (aka, ints or longs) in a DataSeries. The internal representation will be double, despite storing the int or long value. This is how "Volume" works. Volume is a DataSeries, stored as a double, but logically Volume[n] for all values of n is always a whole number, the decimal part should always be zero.

[Edit: Hmm, for the Volume DataSeries, maybe a non-zero value after the decimal is useful for Forex? Ya know, standard lots vs mini lots vs micro lots ... this hierarchy of 'lot' thingies has to be organized somehow, and it all has to be stuffed into the Volume DataSeries somehow, too. Anyways, why Volume is a DataSeries and not a LongSeries is a good question, but I don't know the answer. I know futures really well. But Forex? Not so much. NT8 should be much better for Forex than NT7, my knowledge of Forex will grow as I grow into NT8.]

[Edit-2: On second thought, Volume is a DataSeries on purpose, and not a LongSeries, because that maintains consistency for all the OHLCV parallel array containers being the same, which may have lead to efficiencies in the internal code. But I betcha the main reason is, only the DataSeries object can be passed as input to indicators. Not LongSeries objects, or BoolSeries, just DataSeries objects. DataSeries is extra special this way, and stands above all the other xxxSeries classes for this reason. Thus, ergo, Volume is a DataSeries, not a LongSeries.]

By definition, the class DataSeries stores values of type double. Period.

2. DataSeries can hold multiple distinct values

Yes, one value per bar.

But, by the same token, it cannot hold an arbitrary number of values. The number of values stored in all xxxSeries objects on a particular chart is controlled not by you, but by NinjaTrader.

You are not allowed to have, say, a DataSeries with a maximum of 10 values. Even though you know for a fact that you won't need any values but those 10, there is no way to arbitrarily specify the maximum length of the xxxSeries object.

[Edit: Well, that's not exactly true. You can ask NT to limit the internal array to 256 values, or you can tell NT there is no limit, and the internal array grows to infinity, as needed, to accommodate every bar on your chart, whether that is 500, 5,000, or 5 million. This is the purpose of MaximumBarsLookBack.]

[Edit-2: Even if you could do this, those 10 values would be spread out across 10 bars, one value per bar. If you want to bunch up 10 values for each bar, you would need to think in terms of 10 different DataSeries objects, so that storing values in myDSA[0], myDSB[0], myDSC[0], etc, means they are all sync'ed together, and thus contain your 10 values for the current bar. Your 10 values are related (and stay related) via the index number "n" you specify for the indexer '[n]' when accessing your 10 different DataSeries objects.]

3. DataSeries is mutable

Yes, the values inside the DataSeries are mutable, but once calculated and added to the DataSeries, are rarely changed. Why? Because they are now history, and as more bars are added to the chart, the age of older values increases, meaning the need to reach back and change older values decreases. Think this way, the value at Arr[0] becomes Arr[1], and Arr[1] becomes Arr[2], and so on -- there is a shifting of the indexer '[n]' such that the 'n' is like the age of that data value -- you have little need to change values stored in a xxxSeries object where n > 0.

What I mean is, values being mutable is a rarely used feature of a DataSeries. The value of EMA(24)[10] means the value of EMA(24) at 10 bars ago -- well, that specific value is now an historical value that should not be changed, right? Same with Open[10], or Close[55], etc, these values become older as new bars are added to the chart, and older means historical, and historical means permanent, and permanent means it does not change. No one reaches back into their Close DataSeries and adjusts the close value of the bar 55 bars ago, or even 1 bar ago, there is simply no logical need to do that ... see my point?

But, in other ways, an xxxSeries object is not mutable:

1. A DataSeries can only store values of type double, BoolSeries only values of type bool, etc.

2. You have no control to specify an arbitrary length for the internal array maintained by the xxxSeries object.

3. You have no control when to add new "slots" for values to the DataSeries -- new values are updated in the internal array by you via the Set method, but only 1 new "slot" is added to the internal array per OnBarUpdate.

My sample code showing a "Set" method above (where it just called "Add" to increase the size of the array) is way too simplistic, and technically wrong.

Why? Because, technically, "Set" doesn't add elements, it just updates existing elements that are already there.

[Edit: How did they get there? NinjaTrader adds one new element (aka slot) to each xxxSeries object just before OnBarUpdate. So, by the time you see this xxxSeries object inside your OnBarUpdate code, the new internal element is already there, waiting for you to update it via "Set".]

Calling "Set" 5 times in a row on the same DataSeries (within the same OnBarUpdate context) does not add 5 new items to the internal array -- it updates the same slot (which is usually slot 0, as specified by the indexer '[0]' or just assumed to be slot 0 by Set) each time.

The point is: you cannot add new slots yourself. New slots (aka array elements aka indexer positions) are added to each xxxSeries object automatically just before OnBarUpdate.

These are significant constraints built into the xxxSeries classes. These constraints are by design and on purpose.

So, "mutable" is more in the eye of the beholder.

[bltdavid]

Are you still with me?

Good. Now, let's finalize the DataSeries concept and tie together the terms "slot" and "parallel". Understanding this relationship is the final, most important, concept of a series.

Soon, you'll be able to visualize the internal memory for the internal arrays of the xxxSeries classes, and you'll have extremely good detailed ideas of what the engineers must have done to get all xxxSeries objects to work together.

This final concept is important. In fact, maintaining parallel arrays for the internal arrays of all xxxSeries objects is the most important feature of these classes.

Let's begin. What do we mean by "parallel" arrays?

[Edit: Btw, "parallel arrays" is a real concept, I'm not making it up.]

We mean that for all indexer '[n]' values of n, for all xxxSeries objects associated with the bars of a chart, at any one moment in time, the set of values from all xxxSeries objects at indexer '[n]' remains constant as bars are added to the chart.

That is a mouthful. But it makes sense, the OHLC values of any one bar are the same no matter how old that bar becomes, right? That is, the values of Open[n], High[n], Low[n], and Close[n] will be the same 4 values regardless of how "n" advances when a new bar is added to the chart.

Think of "n" as the "slot" number. It's typically called the BarsAgo index, but it's also helpful to think of the array element positions as slots.

As "n" increases, the set of values for all xxxSeries objects at slot "n" age together, and those values in the arrays at slot "n" do not change.

The idea of parallel arrays is that for each parallel array, the values at slot "n" in all arrays, taken together as a unit, remain the same set of values, at all times, regardless of the value of "n".

Can you see why this must be held true? The OHLC values from, say, 44 bars ago are stored across 4 parallel arrays, and they make up a unit describing that bar, and should not change, right? Now, 10 bars later, that same unit of 4 values is now 54 bars ago, but the same exact 4 OHLC values from the 4 parallel arrays must still line up, same as they did before the 10 new bars were added. See that? As new bars are added to the chart, all the old values become older, they all age together.

That is why you cannot add your own slots to a DataSeries. The slot at indexer '[n]' is tightly controlled so that it stays in sync with all the other parallel arrays of all the other xxxSeries objects on that chart.

Every array element at indexer, say, '[10]', forms a unit of values -- and that unit stays constant -- as the unit moves to [11], and then to [12], and then to [13], and so on, the values stored as separate array elements across all xxxSeries objects stay the same.

How does the unit of values at indexer [10] move to indexer [11]? When a new bar is added to the chart, just before OnBarUpdate is executed, one new slot is added to all internal arrays of all xxxSeries objects on that chart. NinjaTrader does this for you automatically.

Make sense?

[RobVig]

DataSeries.Set(ThankYou)

Beautiful post / dissertation David. I am clear now.
I really appreciate you taking the time to write this.
Very very well done.

Robert

[bltdavid]

Thanks!

[You are a good grasshoppa, my son. But Sledge, Koganam, and I, we need more T-Rex's. You must continue to grow, grasshoppa, gorge yourself on knowledge, and become a T-Rex. Use the Force, Luke. Read, study, expand your horizons. Learn, I say, grow and learn, and join us at the round table of Yoda's and T-Rex's as we both polish and dissect the lovely NinjaTrader. Yoda says, "novice you are, expert you must become". Ok, it's Friday night, I think I better put down the beer, lol, and go back to my movie.]

One last thing.

A "DataSeries" (no space) is a class, provided by the NinjaScript framework.

But a "Data Series" (note the space) is just NT's way of talking about aggregate price data that is added to a chart, you know, the OHLCV values of each bar.

Ok, new assignment for you.

Now that you know about the internal nature of the xxxSeries classes, read this next link carefully, paying attention to the details and relating the high level descriptions of the price data back to the low level DataSeries containers. Ready?

Very carefully study this link:

http://ninjatrader.com/support/helpGuides/nt7/?working_with_price_data.htm

Your mission: Make sure you are able to envision the 5 primary DataSeries objects containing the OHLCV values for each Data Series added to the chart.

[bltdavid]

And, when you're done with that one, study this link,

http://ninjatrader.com/support/helpGuides/nt7/?working_with_price_data2.htm

And then your training will be complete.

Peace out.

[bltdavid]

Oops. Forgot one.

To be a T-Rex, aka an expert, aka a DataSeries Yoda, you'll need to understand this link, too:

http://ninjatrader.com/support/helpGuides/nt7/?valid_input_data_for_indicator.htm

[RobVig]

Yes. I found those same pages / topics and have studied them both. Truth be told, I'm addicted (OCD probably) to this whole programmatic trading thing... I've been thinking about Algorithmic trading for years but thought it was a secret society black box skull and bones secret handshake kinda thing only found in New York and Chicago financial firms and that I'd never get to be apart of. When I discovered NT and NinjaScript and that I could extend it (Indicators and Strategies) I flipped out... Coding into the we hours like never before. Note the timestamp on this post...

HelloNinjaTrader()

[bltdavid]

I'm addicted (OCD probably) to this whole programmatic trading thing...

Have you found this yet?

http://ninjalaunchpad.com/

[koganam]

No, your second assignment will override the value stored by the first assignment.

[Edit: Unless each assignment is happening in a separate invocation of OnBarUpdate, because then the most recent bar accessed by MyDataSeries[0] is actually a new bar, because OnBarUpdate is called when that new bar closes, so the indexer '[0]' is automatically updated under the hood (for free) to always reference the most recently closed bar.]

[Edit-2: Spoiler alert: The above is a convoluted way of saying "slot". Keep reading, "slot" will be introduced soon, and then come back here later and read again after you've achieved a good feel for "slot".]



Yes, they are equivalent -- but isn't it obviously so? I mean, once you assign a value to a variable, then naturally the variable on the left of the equals sign is equivalent to the value on the right of the equals sign.



You would only choose an xxxSeries object over a simple typed object if you have values that need to re/calculated and remembered for every bar.

But, also, because you get some additional memory management for free. I mean, each and every DataSeries (or any instance of an 'xxxSeries' class) has exactly one value maintained for each bar on your chart. These values are maintained in parallel, such that literally any xxxSeries object accessed with suffix [0] always means the value associated with the current bar, any xxxSeries object accessed with suffix [1] always means the value 1 bar ago, and so on.

[Edit: When NinjaTrader wrote their xxxSeries classes, they effectively have to shift all values in the "array" one position to the left, to make the indexer '[0]' free for the new value from the new bar. That's what I mean by NT's DataSeries memory management: NT makes room for the new value at '[0]' for all xxxSeries objects attached to the chart just before calling OnBarUpdate -- it does this automatically for us.]

[Edit-2: I'm almost positive that no shifting of data actually takes place. I'll cut off my left nut if I'm wrong about this: NT engineers used "indexed properties" for the '[0]' indexer, there is no "shifting" to free up indexer '[0]'. See my additional answers below.]

But the real beauty is a bit more complex, and under the hood. The engs at NinjaTrader made each and every indicator a DataSeries. Think about an EMA, or Stochastics. They all have values that are calculated and stay associated with each bar, right?

Consider that EMA(10)[0] is accessing the value of the EMA(10) for the current bar, and that EMA(10)[10] is accessing the value of EMA(10) exactly 10 bars ago. The suffix of '[0]' and '[10]' are indexers, and this is DataSeries syntax applied to an indicator.



Exactly. They do look like an array, the "[ ]" is known as the indexer, and you are exactly right to see that the indexer specifies an age based value (not time, think age). NinjaTrader calls this age based index the BarsAgo index, which makes perfect sense.

[Edit: Why age? Because it is a better metaphor. Open[3] is the bar 15 minutes ago on 5 minute bars, or 3 days ago for daily bars, but what if each bar is a 500-Tick bar, or a 1000-Volume bar, or a Renko bar? Time as a metaphor breaks down, but age continues to be completely correct in its meaning -- here, age literally means n bars ago.]



Sure, you could do this. But you'd be re-inventing so much of the wheel that has been provided to you that, well, it would be nuts. I say that kindly, and with respect.

I mean, consider that NinjaTrader engs could have designed a single class structure to contain all data for each bar inside it, and then made a DataSeries of these objects, such that each instance of TradingDay stored in the DataSeries would represent a separate bar ... sorta like where your code is heading above.

But this design becomes cumbersome later. Why? Because some new values come along that are also associated with each bar -- and logically they should be stored in this class, right?

NT engs decided that, from the point of view of extensibility, multiple parallel arrays containing just one piece of data were better than a single array of a monolith structure that contained all the data. Why? Because it's easier to manage, and much easier to extend -- so they focused on making sure that Open[0] is treated specially to mean the open price of the most recently closed bar, and EMA(24)[0] means the value of EMA(24) of the most recently closed bar, and so on. The 'parallelness' accessed via the indexer '[0]' is maintained by NinjaTrader, for free.

Let's say you write a MillionDollar indicator that takes 3 arguments, x, y, z, so it would be called like MillionDollar(x, y, z) -- but because every indicator is also a DataSeries, we can also use an indexer to say MillionDollar(x, y, z)[n] to access the MillionDollar(x, y, z) value n bars ago.

Consider that the success of third party product called Bloodhound is predicated (partially) on this DataSeries data structure -- the multiple parallel arrays of simple values VS the single array of large class of values -- the engineers at NinjaTrader made some wise choices.

If some other engineer has gone to the trouble of giving me a whole lot of synchronized collections by me merely declaring them, why would I go to the bother of rewriting all that glue needed to put the information into a monolithic class?

Do not get me wrong; NT is by no means perfect and I have cavilled at a number of the NT design decisions, but all told NT is a superbly extensible platform. Why else would a known, picky and cantankerous curmudgeon like me be singing its praises and telling everyone that I no longer code trading solutions in anything else. If you want my code, you better switch to NT! lol.

[sledge]

If some other engineer has gone to the trouble of giving me a whole lot of synchronized collections by me merely declaring them, why would I go to the bother of rewriting all that glue needed to put the information into a monolithic class?

Do not get me wrong; NT is by no means perfect and I have cavilled at a number of the NT design decisions, but all told NT is a superbly extensible platform. Why else would a known, picky and cantankerous curmudgeon like me be singing its praises and telling everyone that I no longer code trading solutions in anything else. If you want my code, you better switch to NT! lol.

It's a simple no-brainer to use. 1 update, 1 set, and you are good to go.

[koganam]

It's a simple no-brainer to use. 1 update, 1 set, and you are good to go.

The question still remains. If all the collections are synchronized, and such synchrony is flawlessly maintained, what is the point in amalgamating the collections with new glue that you have to maintain? Then every time you process that data, you are loading extra data that you may not need for anything. What is the point of creating a single object from objects that are already in synchrony, and can be accessed with minimal resources?