Most effective way to use Data Series?

[jeronymite]

In NT7, I add 56 data series (28 major FX pairs, Ask and Bid, at 1 second interval). This allows me to ensure I can trade with a maximum granularity of 1 second. However, it has drawbacks:

• Data series intensive, and therefore memory and performance intensive
• Limits the programmatic use of indicators to 1 second timeframes

Ideally, what I want is:

• All 28 pairs (Ask and Bid) tradeable with a granularity of no more than 1 second
• Ability to programmatically use indicators at any timeframe, not just those timeframes available in added Data Series. For example, with the Data Series defined as mentioned above, I want to be able to obtain ATR, EMA, RSI etc for 5 minute timeframes without adding additional Data Series (another 56 of them!) at 5 minute intervals.

What would be the most effective and efficient recommended approach in NT8?

Of course, if my approach in NT7 can also be improved, please do not hesitate to advise.

Thanks.

[NinjaTrader_ChelseaB]

Hello jeronymite,

What are describing would be a way to request any data for a time period without having to add the script. (Bars Request)

In NinjaTrader 7, this is not supported to do, but is possible. For an example see the Pivots indicator and the GetBarsNow() method.

In NinjaTrader 8, this will be supported to do, and will be done with the BarsRequest() NinjaScript method.

[jeronymite]

Most effective way to use Data Series?

Thanks, Chelsea. I will look into both of these, for NT7 and NT8.

[jeronymite]

Most effective way to use Data Series?

OK, so let's try NT7 first and then see if I can do the equivalent in NT8 after that. I will need a 1:1 correlation.

So, in NT7, I'm trying to load 5 minute Bid bars (I am using $EURUSD as the instrument in my testing):

private Bars VBs = null ;
private void GetVBs(Instrument inst,
                    int        period,
                    bool       AskBid)
{
    VBs = Data.Bars.GetBars(inst,
                new Period(PeriodType.Minute, period, AskBid ? MarketDataType.Ask : MarketDataType.Bid),
                Bars.From.AddDays(-2),
                Bars.To,
                (Session) Bars.Session.Clone(),
                Data.Bars.SplitAdjust,
                Data.Bars.DividendAdjust) ;
        
    Print ("Exiting with "+VBs.Count+" bars") ;
}
        
private double GetVBsATR()
{
    Print ("Returning " + ATR(VBs,14) + " = " + ATR(VBs,14)[0].ToString("0.00000000")) ;
    return ATR(VBs,14)[0] ;
}

...
GetVBs(BarsArray[1].Instrument,5,false) ;
GetVBsATR() ;
...

The GetVBs Print statement shows more than 600 bars obtained.

But when I invoke GetVBsATR, the following error occurs (in trace.txt):

2015-05-20 10:11:00:080 (NT FXCM) Data.Bars.GetBarsNow: instrument='$EURUSD' from='2015-05-20' to='2015-05-20' period=1 Min - Bid splitAdjusted=False dividendAdjusted=False bars=0 session='Forex'
2015-05-20 10:11:00:082 in OnUnhandledThreadException
2015-05-20 10:11:00:083 *************** unhandled exception trapped ***************
2015-05-20 10:11:00:083 Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
2015-05-20 10:11:00:083    at System.Collections.ArrayList.get_Item(Int32 index)
   at NinjaTrader.Data.DataSeries.get_Item(Int32 barsAgo)
   at NinjaTrader.Indicator.IndicatorBase.get_Item(Int32 barsAgo)
   at NinjaTrader.Strategy.Strategy.GetVBsATR()
   ...
   ...

I note that the trace file shows the GetBarsNow request, apparently completed successfully; but, it says it was a request for 1 minute bars. Strange?!

Your advice most appreciated, as always. Thanks!

[NinjaTrader_Matthew]

Thanks for providing the example. We're deprecating the use of the GetBars() method in NinjaScript (although it is still there) - our development team feels most if not all use cases that may have used GetBars() can be solved with AddDataSeries(). Even our Pivots indicator has been updated to make use of AddDataSeries() and the new session iterator to perform calculations.

In cases where AddDataSeries would not work, you can use a BarsRequest. I'm working on a sample to provide to you to accomplish your goal.

Can you provide me a working script (feel free to remove any proprietary OnBarUpdate() logic, etc... I'd just want to make sure the new approach does meet your request 1:1)

[jeronymite]

Most effective way to use Data Series?

Many thanks, Matthew -- really appreciate your assistance with this.

I need to do this in both NT7 (current code) and NT8 (future migrated code).

The requirement is relatively straightforward. I need to be able to invoke indicator methods to obtain values for DataSeries that have not been added in Initialize. So, for example, in NT7, I currently add various "fast" (1 sec) DataSeries for Ask and Bid for the 28 major currency pairs, giving a total of 1 (default series) + 28 * 2 = 57 DataSeries after Initialize. Orders are managed on these for fast order execution.

I then want to be able to use indicator values for various "slow" DataSeries of the Ask and Bid of the 28 pairs. So, without limitation, since I provide a UI that allows on demand time periods, I need indicator values for longer time periods for which DataSeries have not been added; for example, 1 min, 5 mins, 12 mins, 72 mins, etc.

Hopefully, that explains the requirement well enough. Please let me know if more information or explanation is needed.

Thanks, Matthew!

[NinjaTrader_Matthew]

I guess taking a step back I'm not quite sure why you need to add the 1 second time frame to begin with.

Why can't you just add the desired larger bars period type? A 1-second bars does not update in real-time any fast than a 72-minute, if you're using Calculate.OnEachTick. If you need to slow down OnBarUpdate calls, you can separate logic using IsFirstTickOfBar.

If this was for backtesting reasons in NinjaTrader 7, the problem is now solved with the Tick Replay feature which will give intrabar granularity for your OnBarUpdate method and order fill resolution. However for running a strategy live, I do not see how adding a 1-second *and* a 5/12/72 minute helps in terms of 'faster' execution.

Perhaps I'm missing something... which is why I was hoping you had an sample to demonstrate.

At any rate, taking your approach is doable - at what point do you plan on calling/adding the other time frame to calculate your indicators?

[jeronymite]

Most effective way to use Data Series?

First, very happy for any enlightenment in the NT7 context and better ways to achieve the desired outcome -- thanks!!

As for the rationale and functionality:

• This is for live trading, not backtesting
• I want to add appropriate DataSeries in Initialize that will allow fast trade management for the 28 pairs: Buy/Sell/Close.
• In this strategy, OnBarUpdate is empty because I have developed a custom Winforms UI that facilitates trade management on demand
• There is no requirement to have any charts open (although obviously there would be some)
• The UI allows users to change settings that are used to calculate trade parameters such as position size, stoploss, etc per trade
• The settings include durations used with invoked indicator methods to calculate various parameters in realtime; e.g. use the indicator value for ATR that one would obtain from a 23 minute chart ... or a 39 minute chart or any other duration, using the duration setting as at the time of trade submission
• Since the durations can be changed on demand, one must be able to obtain the values from indicator methods that does not rely on DataSeries added in Initialize
• Currently, I synthesize a series of "virtual bars" using the 1 second DataSeries ... which is both compute intensive and reliant on an awareness of many factors such as session data etc -- being able to load a "new" ad hoc DataSeries is far preferable, being simpler and more reliable

I hope that helps. Thanks!

[jeronymite]

Most effective way to use Data Series?

Hi Matthew. Just wondering if you have been able to progress this? It will certainly help my migration to and testing of NT8B2 if I can solve the problem with the revised NT7 approach.

And just to summarize the situation in NT7 at this stage, I can successfully use GetBars to obtain the bars I need. The issue I currently face is how to use that Bars object to get indicator method values for the obtained bars. So, for example, use GetBars to obtain Bars MyBars. Then how does one do the equivalent of double MyATRvalue = ATR(MyBars,14) or ATR(MyBars,14)[8] ?

Many thanks!

[NinjaTrader_Matthew]

Thanks for checking in - been pretty busy with B3 related tasks, but I'll see if I can get you a working example today.

[jeronymite]

Most effective way to use Data Series?

Hi Matthew. Just checking in again. Hoping you may have time to provide a working example of using the returned Bars object from GetBars to use as the data for any indicator. Fully understand the emphasis on B3, so no problem. Just hoping for some code to be able to move my migration forward in this area. Thanks!

[NinjaTrader_Matthew]

Hi jeronymite, I did work on this, but ran into some complexities which ultimately ended up in a discussion with our development team which opened a hole without a bottom. While it is possible to do this in the NinjaTrader 8 framework, since you now have direct access to a bars series using a BarRequest, there is more involved than I had anticipated:

- Setup the required bars properties
- Request the bars
- Add the request to an array
- Synchronize to the current bar & Account for any multi time frame synchronization
- Handle the states of the target indicator method

The code involved to handle the bars to a series which could be used for an existing indicator input is quite involved.

At this time, our development team advised the product management team not to provide any sort of support in this area, their argument being that since we're so early in beta, the routines used to handle everything above may change down the road, breaking any sort of advice we provide. We've already seen a number of internal changes to the way bars and indicators are handled, which means any suggestions we give at this time will likely change in the near future.

As a result, the most effective and safest way to use a data series is through the current AddDataSeries() method which is only available in an existing NinjaScript's State.Configure.

3rd party developers can feel free to explore implementation based on the BarsRequest class which is our recommended way to get started (using GetBars() is not flexible enough, and those methods are ultimately depreciated). However do be advised that any work in this area is subject to change while we hammer out beta.

For example, you could create your own OnBarUpdate handler like in the example in the BarsRequest I linked above, and re-implement indicator calculations within that handler. However, specific to your request, the process of requesting bars on an existing NinjaScript and then adding those bars as input to an existing indicator in the custom project using the inputs you're use to is just not something we can support at this time.

We do plan on revisiting this area once the platform is in a more stable period (e.g., post release 1).

[jeronymite]

Most effective way to use Data Series?

Many thanks indeed, Matthew, to you and the Development Team. I do appreciate your investigations into this very much.


So, if I'm correct in my understanding of your response, in respect of NT8 NinjaScript:

• This is possible using the AddDataSeries() method but only in State.Configure
• There is continuing work under the covers to make this possible in other States, but that work is a moving target and so not supportable at this time, although after Release 1, it will be revisited and further States may became supported
• Using the BarsRequest class is the recommended approach, although that too is subject to change under the covers whilst in beta
• The ability to use the requested bars as input to indicators in the standard way one would expect with any DataSeries is not supported at this time, but will be addressed after Release 1.

What about in NT7? If I'm reading this correctly, NT7 is not mentioned. Assuming I can obtain the Bars using GetBars, I presume this is the equivalent of the first two steps you list. It would then remain to do the remaining three steps to be able to achieve the goal of a "DataSeries equivalent" BarsArray that one could use as input to indicators. Is that right? And if so, would it be possible to advise on that in the stable NT7 context, please, as a prelude to a migration using the beta-supported NT8 approaches and subsequent post-Release 1 supported methods you have detailed?

Thanks!

[NinjaTrader_Matthew]

To be clear, we will revisit this around that time, I do not know when/if we can technically support this concept.

This concept is not supported in NinjaTrader 7 and we would not spend any time trying to help provide an unsupported solution.

[jeronymite]

Most effective way to use Data Series?

Thanks for the update, Matthew, and for the time spent considering this.


Just so the concept is clear and simple, the need is to be able to:

• Request any DataSeries at any time in any State
• Be able to use that DataSeries in any context where DataSeries can be used, including as input in the normal manner to any indicator or custom indicator
• Be able to remove DataSeries at any time too

In other words, the ability to add and remove DataSeries in a universal context. A second element would be to:

• Do the same as above, but with the DataSeries not being Added, but rather being available to use as an ad hoc request of data stored in a Strategy-specific variable

Many thanks!