How to pass Primary DataSeries to Custom Method

[aventeren]

Then there is a fundamental misunderstanding here of the juxtaposition of time and price. You are overthinking things. Synching is automatic, because it depends only on time. You check the larger time frame for a trigger, and execute the order on the 1-range bars.

IOW whatever happens after you receive a trigger happens after you receive a trigger. You are simply executing on a smaller bar series, because the smaller bar series closes its bars much more frequently than the larger time frame bars, and hence provides a barUpdate more frequently (i.e., effectively, every bar update on the 1-range barSeries is an intrabar update on some bar of the larger time frame series). The issue is not on which bar it is related, the only issue is: "when is the trigger; when is the order". The order will always be subsequent to the trigger. You just need to use the proper BarsInProgress index to differentiate which barUpdate event you are handling.

That makes a ton of sense. Thanks.

I'll work on that and post up what I ended up doing. My hope is that others will come across this thread and not have to crawl the backtesting learning curve like I am currently doing. To that end, I thank you for your help.

There is a saying in Spanish that says "Poco a poco se va legos", which translates roughly to "Little by little, one goes far." Clearly that's where I'm at.

Thanks a ton.

All best,

Aventeren

[koganam]

His point was that Range and Tick bars are not synced to time charts, so running a 1 Range chart on a time chart is impossible because the secondary series would not know where to pick start and stop in the context of the primary bar--unless of course you programmed in some ToTime() functions to ensure that the secondary series was only operating within the boundaries of the primary bar.

I sure would be interested in seeing a primary time chart (minute, day, week, etc) coupled with a 1 Range secondary bar for back testing, but heretofore I have been unable to see one in the wild. That's what I was trying to do with my FindTriggerPrice() Method, but alas I have hit a wall.

To the extent you would be willing to share an example, I'd love to see it.

Thanks,

Aventeren

Then there is a fundamental misunderstanding here of the juxtaposition of time and price. You are overthinking things. Synching is automatic, because it depends only on time. You check the larger time frame for a trigger, and execute the order on the 1-range bars.

IOW whatever happens after you receive a trigger happens after you receive a trigger. You are simply executing on a smaller bar series, because the smaller bar series closes its bars much more frequently than the larger time frame bars, and hence provides a barUpdate more frequently (i.e., effectively, every bar update on the 1-range barSeries is an intrabar update on some bar of the larger time frame series). The issue is not on which bar it is related, the only issue is: "when is the trigger; when is the order". The order will always be subsequent to the trigger. You just need to use the proper BarsInProgress index to differentiate which barUpdate event you are handling.

[aventeren]

I do not know why you were told that, and it may well be true, but not in my experience. I use a 1-range secondary series rather than a 1-tick one for the simple reason that the former is a closer simulation to reality, where the market must usually trade through a price before it fills the order. Moreover, it is far less processor-intensive to use a 1-range series instead of a 1-tick series, where multiple ticks will come in at the same price on a really liquid equity.

My signals are taken from the higher time frame series and executed on the 1-range series. Granted that a lot of my automated (and manual) trading strategies depend on Range Bars, I have also optimized on time-bars.

His point was that Range and Tick bars are not synced to time charts, so running a 1 Range chart on a time chart is impossible because the secondary series would not know where to pick start and stop in the context of the primary bar--unless of course you programmed in some ToTime() functions to ensure that the secondary series was only operating within the boundaries of the primary bar.

I sure would be interested in seeing a primary time chart (minute, day, week, etc) coupled with a 1 Range secondary bar for back testing, but heretofore I have been unable to see one in the wild. That's what I was trying to do with my FindTriggerPrice() Method, but alas I have hit a wall.

To the extent you would be willing to share an example, I'd love to see it.

Thanks,

Aventeren

[koganam]

I really appreciate your perspective, koganam. Perhaps I am overthinking this.

I was told by NT Support (Support Thread Here") that to use a 1 Range secondary series that you then had to use a matched higher time frame ("HTF") primary series--so in this case a (1 + X) Range primary series. Other examples would be a 800 tick - 1 tick, 60 min - 1 min, 15 Range - 1 Range, etc.

So when you use your 1 Range secondary, are you also using a Range primary?

Have you found a way to use a 1 Range secondary series to backtest a non-Range primary series? If so, would you mind dropping a few hints?

Again, I really appreciate your perspective, koganam.

Thanks,

Aventeren

I do not know why you were told that, and it may well be true, but not in my experience. I use a 1-range secondary series rather than a 1-tick one for the simple reason that the former is a closer simulation to reality, where the market must usually trade through a price before it fills the order. Moreover, it is far less processor-intensive to use a 1-range series instead of a 1-tick series, where multiple ticks will come in at the same price on a really liquid equity.

My signals are taken from the higher time frame series and executed on the 1-range series. Granted that a lot of my automated (and manual) trading strategies depend on Range Bars, I have also optimized on time-bars.

[aventeren]

I backtest intrabar by using a secondary 1-range barSeries. However, I hardly ever actually test intrabar. I find such fine-grained testing unnecessary, as a retail trader. If I were designing HFT, it would be a different kettle of fish. For retail trading purposes, as far as I am concerned, a bar speaks only when it closes. I have seen too many times when conditions whip back and forth, creating numerous losses for Strategies that do not wait for a bar to close.

In fact, because of this, despite "the customer being always right", I generally refuse to create Strategies that do not trade on barClose. They take extremely long to optimize, and then when I actually run statistics on them, I discover that they are often just tightly curve-fit on a lookahead bias, and unviable.

Every Strategy that I have supplied to brokerage auto-trading partners enforces COBC=close. They can change it to false: I simply return it to true.

I really appreciate your perspective, koganam. Perhaps I am overthinking this.

I was told by NT Support (Support Thread Here") that to use a 1 Range secondary series that you then had to use a matched higher time frame ("HTF") primary series--so in this case a (1 + X) Range primary series. Other examples would be a 800 tick - 1 tick, 60 min - 1 min, 15 Range - 1 Range, etc.

So when you use your 1 Range secondary, are you also using a Range primary?

Have you found a way to use a 1 Range secondary series to backtest a non-Range primary series? If so, would you mind dropping a few hints?

Again, I really appreciate your perspective, koganam.

Thanks,

Aventeren

[koganam]

Just to confirm, if I my triggerDataSeries contains historical values equal to the corresponding bar indexed Close values (i.e., I Set triggerDataSeries[0] to Close at the end of OnBarUpdate) and a triggerDataSeries[0] value equal to a temporary "test" value, when I pass the triggerDataSeries to the NT EMA indicator, the value I receive back should be equal to what the EMA would be with the "test" value (i.e., triggerDataSeries[0] was Set to "test" value) and the historical Close values, correct?

IOW, the EMA doesn't care what DataSeries you give it, because it will calculate an EMA based on those values--and therefore the EMA can be passed a DataSeries type and does not require an IDataSeries type?

I'm seeing that there is a distinct difference between a DataSeries type (which just holds one bar indexed value) and an IDataSeries type (which appears to hold the bar indexed OHLC values). And furthermore, that many of the stock NT indicators require IDataSeries type inputs, which DataSeries are decidedly not?

Can someone please conclusively state that an IDataSeries type is NOT a DataSeries type? It just seems to me that they share the words "DataSeries"--but that their only common feature is that they are bar indexed, and that a DataSeries type will not work as passed parameter to an indicator that requires an IDataSeries type?

Boy we're in the weeds. I hope this isn't getting confusing. Am I overthinking this?

Thanks,

Aventeren

Yes, you are overthinking it, unfortunately. IDataSeries is an Interface to a DataSeries. Every indicator in NT can take a DataSeries as a parameter.

[koganam]

Thanks, koganam, I'll have to think about this. The challenge is that I am trying to first find the actual price that caused the cross to occur, and then I am trying to use that price to calculate what the ATR would have been at that price. Given my goal of wanting to create a sound backtest, I'm beginning to believe that the only way I'll be able to do this is by creating synthetic OHLC DataSeries variables that are then passed to a custom ATR Method that takes the synthetic OHLC DataSeries variables and calculates the ATR. As of right now, my triggerDataSeries only tracks the Close values--and not the OHL--which makes calculating the ATR impossible. So I need to fix this...

I've been trying to avoid re-writing each indicator I backtest, but clearly in some cases I'll have to.

The frustrating thing is that I'll have two different strategies to test the same idea: one for backtesting and one for forward testing--because with forward testing I'll be able to use the stock indicators.

The deeper I get into this backtesting wormhole, I'm surprised that there is not wider consensus on how to code an intrabar strategy backtest. Maybe I just don't understand how to properly use the lower time frame secondary series (i.e., 1 min) to create sound backtest results.

How are you backtesting your strategies? How would you backtest the simple MACross strategy? (Side note, clearly I'm not interested in trading the MACross, I'm just trying to use it to understand how to create a sound backtest).

Thanks for your help!

Aventeren

I backtest intrabar by using a secondary 1-range barSeries. However, I hardly ever actually test intrabar. I find such fine-grained testing unnecessary, as a retail trader. If I were designing HFT, it would be a different kettle of fish. For retail trading purposes, as far as I am concerned, a bar speaks only when it closes. I have seen too many times when conditions whip back and forth, creating numerous losses for Strategies that do not wait for a bar to close.

In fact, because of this, despite "the customer being always right", I generally refuse to create Strategies that do not trade on barClose. They take extremely long to optimize, and then when I actually run statistics on them, I discover that they are often just tightly curve-fit on a lookahead bias, and unviable.

Every Strategy that I have supplied to brokerage auto-trading partners enforces COBC=true. They can change it to false: I simply return it to true.

[aventeren]

EMA calls with DataSeries?

Just to confirm, if I my triggerDataSeries contains historical values equal to the corresponding bar indexed Close values (i.e., I Set triggerDataSeries[0] to Close at the end of OnBarUpdate) and a triggerDataSeries[0] value equal to a temporary "test" value, when I pass the triggerDataSeries to the NT EMA indicator, the value I receive back should be equal to what the EMA would be with the "test" value (i.e., triggerDataSeries[0] was Set to "test" value) and the historical Close values, correct?

IOW, the EMA doesn't care what DataSeries you give it, because it will calculate an EMA based on those values--and therefore the EMA can be passed a DataSeries type and does not require an IDataSeries type?

I'm seeing that there is a distinct difference between a DataSeries type (which just holds one bar indexed value) and an IDataSeries type (which appears to hold the bar indexed OHLC values). And furthermore, that many of the stock NT indicators require IDataSeries type inputs, which DataSeries are decidedly not?

Can someone please conclusively state that an IDataSeries type is NOT a DataSeries type? It just seems to me that they share the words "DataSeries"--but that their only common feature is that they are bar indexed, and that a DataSeries type will not work as passed parameter to an indicator that requires an IDataSeries type?

Boy we're in the weeds. I hope this isn't getting confusing. Am I overthinking this?

Thanks,

Aventeren

[aventeren]

ATR() is calculated from Highs, Lows and Closes, so will not be the same as an average travel calculated from single values.

If you want to calculate ATR(), then reference that bar series that you want to calculate from, not your internal DataSeries. IOW, either use the BarsInProgress fillter with the ATR(), or use the Closes[x][1], Lows[x][0], Highs[x][0] and calculate the ATR using a custom method.

Thanks, koganam, I'll have to think about this. The challenge is that I am trying to first find the actual price that caused the cross to occur, and then I am trying to use that price to calculate what the ATR would have been at that price. Given my goal of wanting to create a sound backtest, I'm beginning to believe that the only way I'll be able to do this is by creating synthetic OHLC DataSeries variables that are then passed to a custom ATR Method that takes the synthetic OHLC DataSeries variables and calculates the ATR. As of right now, my triggerDataSeries only tracks the Close values--and not the OHL--which makes calculating the ATR impossible. So I need to fix this...

I've been trying to avoid re-writing each indicator I backtest, but clearly in some cases I'll have to.

The frustrating thing is that I'll have two different strategies to test the same idea: one for backtesting and one for forward testing--because with forward testing I'll be able to use the stock indicators.

The deeper I get into this backtesting wormhole, I'm surprised that there is not wider consensus on how to code an intrabar strategy backtest. Maybe I just don't understand how to properly use the lower time frame secondary series (i.e., 1 min) to create sound backtest results.

How are you backtesting your strategies? How would you backtest the simple MACross strategy? (Side note, clearly I'm not interested in trading the MACross, I'm just trying to use it to understand how to create a sound backtest).

Thanks for your help!

Aventeren

[koganam]

Backtesting. My goal is to create effective backtests--and particularly as if COBC would be equal to false (i.e., intrabar orders can be placed) when I forward sim and live trade the strategy. Therefore, if my intention is to trade intrabar when I start trading sim and live, I must also create a backtest that would also execute orders intrabar.

To date, I have taken the MACross strategy and attempted to first locate a signal bar that triggered the cross, and then via the FindTriggerPrice() Method I am attempting to find the individual price within the signal bar that would have caused the cross in live trading--which would have been the price that the live strategy would have fired an order.

Because I am testing various prices within the FindTriggerPrice() Method against, in this case, the EMA, I created a DataSeries called triggerDataSeries that holds these testing values, and then triggerDataSeries is used within the EMA calculation. If the EMA calculation signals a cross, I then know that triggerDataSeries[0] was the triggering price and I use that value as my limit order entry price.

So the FindTriggerPrice() and EMACross() Methods do nothing but test various prices to identify the limit entry price.

Once I have identified a limit entry price, I also need to calculate and set my stop and profit targets, which I would like to be ATR based. Therefore, because the current value of the triggerDataSeries (which is triggerDataSeries[0]) still holds the trigger price where the cross initially occurred, I need the ATR to be calculated with the triggerDataSeries being the high or low of the bar (depending if the cross is long or short, which the code already does). But the challenge, as you have identified, with passing the triggerDataSeries to the ATR function, is that triggerDataSeries is of type DataSeries--not type IDataSeries, which presumably would hold the OHLC of each bar. Therefore, I would need to have a way of creating a synthetic IDataSeries parameter to send to ATR--but given that synthetic IDataSeries parameters are not supported, I'm not sure how I'm going to skin this cat.

So this is all about backtesting.

I appreciate your help thus far, and I remain hopeful that we'll get this figured out.

Thanks!

Aventeren

ATR() is calculated from Highs, Lows and Closes, so will not be the same as an average travel calculated from single values.

If you want to calculate ATR(), then reference that bar series that you want to calculate from, not your internal DataSeries. IOW, either use the BarsInProgress fillter with the ATR(), or use the Closes[x][1], Lows[x][0], Highs[x][0] and calculate the ATR using a custom method.

[aventeren]

Hi aventeren,

What are you trying to accomplish with this?

There may be another way to go about it.

Backtesting. My goal is to create effective backtests--and particularly as if COBC would be equal to false (i.e., intrabar orders can be placed) when I forward sim and live trade the strategy. Therefore, if my intention is to trade intrabar when I start trading sim and live, I must also create a backtest that would also execute orders intrabar.

To date, I have taken the MACross strategy and attempted to first locate a signal bar that triggered the cross, and then via the FindTriggerPrice() Method I am attempting to find the individual price within the signal bar that would have caused the cross in live trading--which would have been the price that the live strategy would have fired an order.

Because I am testing various prices within the FindTriggerPrice() Method against, in this case, the EMA, I created a DataSeries called triggerDataSeries that holds these testing values, and then triggerDataSeries is used within the EMA calculation. If the EMA calculation signals a cross, I then know that triggerDataSeries[0] was the triggering price and I use that value as my limit order entry price.

So the FindTriggerPrice() and EMACross() Methods do nothing but test various prices to identify the limit entry price.

Once I have identified a limit entry price, I also need to calculate and set my stop and profit targets, which I would like to be ATR based. Therefore, because the current value of the triggerDataSeries (which is triggerDataSeries[0]) still holds the trigger price where the cross initially occurred, I need the ATR to be calculated with the triggerDataSeries being the high or low of the bar (depending if the cross is long or short, which the code already does). But the challenge, as you have identified, with passing the triggerDataSeries to the ATR function, is that triggerDataSeries is of type DataSeries--not type IDataSeries, which presumably would hold the OHLC of each bar. Therefore, I would need to have a way of creating a synthetic IDataSeries parameter to send to ATR--but given that synthetic IDataSeries parameters are not supported, I'm not sure how I'm going to skin this cat.

So this is all about backtesting.

I appreciate your help thus far, and I remain hopeful that we'll get this figured out.

Thanks!

Aventeren

[NinjaTrader_ChelseaB]

Hi aventeren,

What are you trying to accomplish with this?

There may be another way to go about it.

[aventeren]

Hi aventeren,

It seems like this has something to do with the high low and open being available when calling with Close.

I am currently looking into this and I will let you know what I find.

Great point. triggerDataSeries only contains the primary DataSeries's Close value, so I don't know how we can use triggerDataSeries to pass to ATR because triggerDataSeries only has the Close values and not the Highs and Lows. (Here is a link to NT's ATR definition).

So I guess the next question to ask: Is there a way to create an actual IDataSeries that contains a synthetic OHLC?

Thanks,

Aventeren

[NinjaTrader_ChelseaB]

Hi aventeren,

It seems like this has something to do with the high low and open being available when calling with Close.

I am currently looking into this and I will let you know what I find.

[aventeren]

Another ATR and Secondary DataSeries Issue Example

So I've been thinking more about this secondary DataSeries issue as it relates to the ATR indicator, and I decided to isolate it further to see if there was something else going on in my code that would cause the difference.

So I created a new Strategy (attached) that does nothing but assign Close[0] to triggerDataSeries via the Set command for every bar on the chart.

I then modified the GetATR() Method to print out the values for each bar's Close[i], triggerDataseries[i], ATR(PeriodATR)[i] and ATR(triggerDataSeries, PeriodATR)[i].

I then put the GetATR() Method call at the end of OnBarUpdate so it was the last thing that was done for each bar after the number of bars was > PeriodATR.

Here are the results from the Output Window:



As you can see from the Output Window, the values of Close[i] and triggerDataSeries[i] are the same. However, the ATR values for the primary (i.e., Close) and secondary (i.e., triggerDataSeries) are much different.

This is a head scratcher for me, and I hope we can figure it out.

I'm standing by to respond to any comments or questions.

Thanks for you help,

Aventeren