How to pass Primary DataSeries to Custom Method

[NinjaTrader_ChelseaB]

Hi Aventeren,

Are there any errors in the Log tab of the Control Center.

You also have TraceOrders set to true. Do you have the output window open?

When the SetStopLoss gets called do you see the print in the output window about it?

If you open chart trader to the same instrument you can see working orders. After entering a position do you see the profit target appear? If you increase the scaling do you see the stop loss?

[aventeren]

Thanks for responding, Chelsea. Here are my responses to your questions.

Are there any errors in the Log tab of the Control Center.

No

You also have TraceOrders set to true. Do you have the output window open?

When the SetStopLoss gets called do you see the print in the output window about it?

Here is an Output window screenshot.



I modified the code with a bunch of Print statements to navigate through the various Methods. I hope they provide some insight.

Also, here is a screen grab of the chart showing that the "short limit" was placed "1 @ 1646.75". The chart also shows the Position.MarketPosition states (i.e. the F (Flat), L (Long) and S (Short) letters). Plus, you can see the DarkRed diamond that a drew on the chart from within OnBarUpdate.



Some observations from the Output Window:

1. The stop and profit target "Price Value=" values do not match the passed and verified values. Why?

2. The entry's "PriceValue=1646.75" appears to be right, but the Position.AvgPrice = 0.00. Why?

If you open chart trader to the same instrument you can see working orders. After entering a position do you see the profit target appear? If you increase the scaling do you see the stop loss?

When I open a new ES 03-14 daily chart with Chart Trader open, I do not see the orders. I tried to see the orders on both of my Sim accounts plus my live account, and nothing. Any ideas?

I have attached the updated strategy with the new print statements.

This has to be something really basic because it seems like my logic is right. Clearly I must be missing something really fundamental about order placement and execution. In any event, I look forward to your comments, questions and ideas.

All best,

Aventeren

[NinjaTrader_ChelseaB]

Hi aventeren,

Take a look at the prints in your output window and take a look at the go short code.

The price from FindTriggerPrice() and set to priceMethod is 1646.75.
The price that comes back from GetStop() is 1662.75.

You then add these together when setting the stop.
SetStopLoss("short limit", CalculationMode.Price, priceMethod + priceStop, false);
1646.75 + 1662.75 = 3309.5

and thats exactly what happens in your output file:

10/9/2013 9:00:00 PM Entered internal SetStopTarget() method: Type=Stop FromEntrySignal='short limit' Mode=Price Value=3309.5 Currency=0 Simulated=False

For the profit target you are subtracting priceMethod - pricePT.
1649.75 - 1614.75 = 32


In other words, the stop loss and profit target are both being set exactly where you have told them. Its just really really far away from the current price.

[aventeren]

Hi aventeren,

Take a look at the prints in your output window and take a look at the go short code.

The price from FindTriggerPrice() and set to priceMethod is 1646.75.
The price that comes back from GetStop() is 1662.75.

You then add these together when setting the stop.
SetStopLoss("short limit", CalculationMode.Price, priceMethod + priceStop, false);
1646.75 + 1662.75 = 3309.5

and thats exactly what happens in your output file:

10/9/2013 9:00:00 PM Entered internal SetStopTarget() method: Type=Stop FromEntrySignal='short limit' Mode=Price Value=3309.5 Currency=0 Simulated=False

For the profit target you are subtracting priceMethod - pricePT.
1649.75 - 1614.75 = 32


In other words, the stop loss and profit target are both being set exactly where you have told them. Its just really really far away from the current price.

Well I'll be diggity damned if that didn't do it. Wow. I knew it had to be something easy. Thanks for the second set of eyes!

Okay so now what do we have here? We have orders being executed at the price that actually caused the cross, but we the order is not executed until after:

1. After the close of the trigger (signal) bar, and
2. If the bar(s) following the trigger (signal) trade through the limit price.

We can see the 1 bar delay by the fact that the diamond is printed on top of the trigger bar but the order arrow is not placed until the next bar (assuming that the next bar traded through the entry price, of course).

I found an example of a trade working correctly and an example of a trade that found the trigger price but then expired because price did not trade through the entry price again. Here is the chart showing these two cases.



And here is a screenshot of the Output Window showing that the trigger price was found and everything was set to go long, but then the order was cancelled.



So to have the order placed intrabar, we must use a secondary series?

If we must use a secondary series, let's assume that I load in a 1 minute series. I would then need to place all of what I have done thus far within BarsInProgress == 0, however I would need to place the order within BarsInProgress == 1. Correct?

As a follow up, then, the chart that I have shown is on a ES Daily chart, but let's assume that it was on 60 min chart with a bar time of 12:05pm, which would be the bar's closing time, which would further mean that the bar would have opened at 12:00:01pm. My question if I load the 1 minute series in: Given that the primary bar (in this case) were to exist in between the times noted, what would what be the first secondary bar referenced? Would it be the 12:01pm bar, which would mean that the bar opened at 12:00:01pm and will close at 12:00pm? And as another follow up, would I just continue calling the entry command using BarsInProgress == 1 until the order is either cancelled (like it was in the above example) or executed by the secondary series?

Is there any way to simulate a primary intrabar entry?

Finally, do the stop and profit targets also behave in this same manner? Is there first a "stop" signal bar and "profit target" signal bar that must first close and then have price trade through them to close the trade or are they assumed to have executed within the stop/profit target signal bar?

Thanks for your continued help, Chelsea. I really appreciate it.

All best,

Aventeren

[aventeren]

Still having secondary DataSeries to ATR issue...

Chelsea--

I figured that I would hold this question until I got things working, but now that I'm moving in the right direction I figured that I needed to come back to it.

If you go to the GetATR() Method and change the code to this:

#region GetATR()
		private double GetATR()
		{
			double	methodATR;
			methodATR	=	ATR(triggerDataSeries, PeriodATR)[0];
			return methodATR;
		}
		#endregion GetATR()

and then recompile the code and reload it in the chart, you'll see that the ATR value drops to sub 11 (again, I'm looking at a daily ES 03-14 chart loaded with 180 days of data...I think previously I may have said it was a 60 min chart...it's a daily chart--my bad). Clearly this is off.

If you track through the triggerDataSeries Set points, the are:

1. If CurrentBar less than the periods, it's Set to Close[0]

2. Within FindTriggerPrice() Method, where it's Set to various values until the trigger occurs, at which point FindTriggerMethod() Returns--which should leave the triggerDataSeries[0] equal to the trigger price.

3. At the end of OnBarUpdate.

I then installed a quick for loop within the GetATR() Method to track what the recorded Close and triggerDataSeries values were, and the for loop shows that all of the historical bars always match, while the current triggerDataSeries[0] generated after the FindTriggerPrice() Method is set to the trigger price instead of the Close price of the bar, so that is even right--because in using the ATR(triggerDataSeries, PeriodATR) I was trying to calculate the ATR as it would have been when the trigger occurred. Yet within the same for loop I also calculated the ATR using the primary DataSeries (by ATR(PeriodATR) and the triggerDataSeries (by ATR(triggerDataSeries, PeriodATR), and the results are wildly different.

So this is weird because the values are correct, but I just can't see how the ATRs would be that different, when 13 of the 14 period ATR are equal while the final (trigger) bar's difference between Close and the trigger price during this period were only 4 to 15 points different. So something weird is going on with NT's calculation of the ATR using a secondary data series--and I can't figure out what it is.

To test and recreate this problem, I've attached V003 of the strategy. Make sure you open the Output Window because I printed the Close and tiggerDataSeries historical and ATR values from the GetATR() Method for loop so you can see the differences.

Hopefully we can figure this out.

I'm standing by with your questions and comments.

Thanks,

Aventeren

[NinjaTrader_ChelseaB]

Hi aventeren,

If an order is submitted when a bar closes, the time of the order is going to be the start time of the new bar. This is why an order that is triggered on one bar shows on the next.

The order in your output that was cancelled was expired. That means that the order was not created with the liveUntilCancelled bool as true. When the liveUntilCancelled bool is not used, by default it is false and this means that any working order such as a limit or stop will be cancelled on the next bar.

To place an order with liveUntilCancelled as true try:
EnterLongLimit(0, true, 1, Low[0]-2*TickSize, "myOrder");
EnterLongLimit(int barsInProgressIndex, bool liveUntilCancelled, int quantity, double limitPrice, string signalName)
http://www.ninjatrader.com/support/h...rlonglimit.htm


If you would like to place an order intrabar, you can set the strategy to Calculate on bar close as false which will cause OnBarUpdate to run every tick.
http://www.ninjatrader.com/support/h...onbarclose.htm

Use if (FirstTickOfBar) { /* execute code */ } to detect when a bar has closed.
http://www.ninjatrader.com/support/h...ttickofbar.htm

You can also use intrabar granularity and process during BarsInProgress 1 but use the primary data series for calculations (BarsArray[0]).

[aventeren]

Hi aventeren,

If an order is submitted when a bar closes, the time of the order is going to be the start time of the new bar. This is why an order that is triggered on one bar shows on the next.

The order in your output that was cancelled was expired. That means that the order was not created with the liveUntilCancelled bool as true. When the liveUntilCancelled bool is not used, by default it is false and this means that any working order such as a limit or stop will be cancelled on the next bar.

To place an order with liveUntilCancelled as true try:
EnterLongLimit(0, true, 1, Low[0]-2*TickSize, "myOrder");
EnterLongLimit(int barsInProgressIndex, bool liveUntilCancelled, int quantity, double limitPrice, string signalName)
http://www.ninjatrader.com/support/h...rlonglimit.htm


If you would like to place an order intrabar, you can set the strategy to Calculate on bar close as false which will cause OnBarUpdate to run every tick.
http://www.ninjatrader.com/support/h...onbarclose.htm

Use if (FirstTickOfBar) { /* execute code */ } to detect when a bar has closed.
http://www.ninjatrader.com/support/h...ttickofbar.htm

You can also use intrabar granularity and process during BarsInProgress 1 but use the primary data series for calculations (BarsArray[0]).

Thanks for the help on why the orders were being cancelled. I'll do some research on what you've provided.

So to have a backtest get us closer to an intrabar backtest, the only way to do this is by placing the orders with the secondary series?

If the answer is yes, a secondary series must be used to get closer to an intrabar backtest, can you also please help me understand the following?

As a follow up, then, the chart that I have shown is on a ES Daily chart, but let's assume that it was on 60 min chart with a bar time of 12:05pm, which would be the bar's closing time, which would further mean that the bar would have opened at 12:00:01pm. My question if I load the 1 minute series in: Given that the primary bar (in this case) were to exist in between the times noted, what would what be the first secondary bar referenced? Would it be the 12:01pm bar, which would mean that the bar opened at 12:00:01pm and will close at 12:00pm? And as another follow up, would I just continue calling the entry command using BarsInProgress == 1 until the order is either cancelled (like it was in the above example) or executed by the secondary series?

Thanks!

Aventeren

[aventeren]

Another ATR and Secondary DataSeries Issue Example

So I've been thinking more about this secondary DataSeries issue as it relates to the ATR indicator, and I decided to isolate it further to see if there was something else going on in my code that would cause the difference.

So I created a new Strategy (attached) that does nothing but assign Close[0] to triggerDataSeries via the Set command for every bar on the chart.

I then modified the GetATR() Method to print out the values for each bar's Close[i], triggerDataseries[i], ATR(PeriodATR)[i] and ATR(triggerDataSeries, PeriodATR)[i].

I then put the GetATR() Method call at the end of OnBarUpdate so it was the last thing that was done for each bar after the number of bars was > PeriodATR.

Here are the results from the Output Window:



As you can see from the Output Window, the values of Close[i] and triggerDataSeries[i] are the same. However, the ATR values for the primary (i.e., Close) and secondary (i.e., triggerDataSeries) are much different.

This is a head scratcher for me, and I hope we can figure it out.

I'm standing by to respond to any comments or questions.

Thanks for you help,

Aventeren

[NinjaTrader_ChelseaB]

Hi aventeren,

It seems like this has something to do with the high low and open being available when calling with Close.

I am currently looking into this and I will let you know what I find.

[aventeren]

Hi aventeren,

It seems like this has something to do with the high low and open being available when calling with Close.

I am currently looking into this and I will let you know what I find.

Great point. triggerDataSeries only contains the primary DataSeries's Close value, so I don't know how we can use triggerDataSeries to pass to ATR because triggerDataSeries only has the Close values and not the Highs and Lows. (Here is a link to NT's ATR definition).

So I guess the next question to ask: Is there a way to create an actual IDataSeries that contains a synthetic OHLC?

Thanks,

Aventeren

[NinjaTrader_ChelseaB]

Hi aventeren,

What are you trying to accomplish with this?

There may be another way to go about it.

[aventeren]

Hi aventeren,

What are you trying to accomplish with this?

There may be another way to go about it.

Backtesting. My goal is to create effective backtests--and particularly as if COBC would be equal to false (i.e., intrabar orders can be placed) when I forward sim and live trade the strategy. Therefore, if my intention is to trade intrabar when I start trading sim and live, I must also create a backtest that would also execute orders intrabar.

To date, I have taken the MACross strategy and attempted to first locate a signal bar that triggered the cross, and then via the FindTriggerPrice() Method I am attempting to find the individual price within the signal bar that would have caused the cross in live trading--which would have been the price that the live strategy would have fired an order.

Because I am testing various prices within the FindTriggerPrice() Method against, in this case, the EMA, I created a DataSeries called triggerDataSeries that holds these testing values, and then triggerDataSeries is used within the EMA calculation. If the EMA calculation signals a cross, I then know that triggerDataSeries[0] was the triggering price and I use that value as my limit order entry price.

So the FindTriggerPrice() and EMACross() Methods do nothing but test various prices to identify the limit entry price.

Once I have identified a limit entry price, I also need to calculate and set my stop and profit targets, which I would like to be ATR based. Therefore, because the current value of the triggerDataSeries (which is triggerDataSeries[0]) still holds the trigger price where the cross initially occurred, I need the ATR to be calculated with the triggerDataSeries being the high or low of the bar (depending if the cross is long or short, which the code already does). But the challenge, as you have identified, with passing the triggerDataSeries to the ATR function, is that triggerDataSeries is of type DataSeries--not type IDataSeries, which presumably would hold the OHLC of each bar. Therefore, I would need to have a way of creating a synthetic IDataSeries parameter to send to ATR--but given that synthetic IDataSeries parameters are not supported, I'm not sure how I'm going to skin this cat.

So this is all about backtesting.

I appreciate your help thus far, and I remain hopeful that we'll get this figured out.

Thanks!

Aventeren

[koganam]

Backtesting. My goal is to create effective backtests--and particularly as if COBC would be equal to false (i.e., intrabar orders can be placed) when I forward sim and live trade the strategy. Therefore, if my intention is to trade intrabar when I start trading sim and live, I must also create a backtest that would also execute orders intrabar.

To date, I have taken the MACross strategy and attempted to first locate a signal bar that triggered the cross, and then via the FindTriggerPrice() Method I am attempting to find the individual price within the signal bar that would have caused the cross in live trading--which would have been the price that the live strategy would have fired an order.

Because I am testing various prices within the FindTriggerPrice() Method against, in this case, the EMA, I created a DataSeries called triggerDataSeries that holds these testing values, and then triggerDataSeries is used within the EMA calculation. If the EMA calculation signals a cross, I then know that triggerDataSeries[0] was the triggering price and I use that value as my limit order entry price.

So the FindTriggerPrice() and EMACross() Methods do nothing but test various prices to identify the limit entry price.

Once I have identified a limit entry price, I also need to calculate and set my stop and profit targets, which I would like to be ATR based. Therefore, because the current value of the triggerDataSeries (which is triggerDataSeries[0]) still holds the trigger price where the cross initially occurred, I need the ATR to be calculated with the triggerDataSeries being the high or low of the bar (depending if the cross is long or short, which the code already does). But the challenge, as you have identified, with passing the triggerDataSeries to the ATR function, is that triggerDataSeries is of type DataSeries--not type IDataSeries, which presumably would hold the OHLC of each bar. Therefore, I would need to have a way of creating a synthetic IDataSeries parameter to send to ATR--but given that synthetic IDataSeries parameters are not supported, I'm not sure how I'm going to skin this cat.

So this is all about backtesting.

I appreciate your help thus far, and I remain hopeful that we'll get this figured out.

Thanks!

Aventeren

ATR() is calculated from Highs, Lows and Closes, so will not be the same as an average travel calculated from single values.

If you want to calculate ATR(), then reference that bar series that you want to calculate from, not your internal DataSeries. IOW, either use the BarsInProgress fillter with the ATR(), or use the Closes[x][1], Lows[x][0], Highs[x][0] and calculate the ATR using a custom method.

[aventeren]

ATR() is calculated from Highs, Lows and Closes, so will not be the same as an average travel calculated from single values.

If you want to calculate ATR(), then reference that bar series that you want to calculate from, not your internal DataSeries. IOW, either use the BarsInProgress fillter with the ATR(), or use the Closes[x][1], Lows[x][0], Highs[x][0] and calculate the ATR using a custom method.

Thanks, koganam, I'll have to think about this. The challenge is that I am trying to first find the actual price that caused the cross to occur, and then I am trying to use that price to calculate what the ATR would have been at that price. Given my goal of wanting to create a sound backtest, I'm beginning to believe that the only way I'll be able to do this is by creating synthetic OHLC DataSeries variables that are then passed to a custom ATR Method that takes the synthetic OHLC DataSeries variables and calculates the ATR. As of right now, my triggerDataSeries only tracks the Close values--and not the OHL--which makes calculating the ATR impossible. So I need to fix this...

I've been trying to avoid re-writing each indicator I backtest, but clearly in some cases I'll have to.

The frustrating thing is that I'll have two different strategies to test the same idea: one for backtesting and one for forward testing--because with forward testing I'll be able to use the stock indicators.

The deeper I get into this backtesting wormhole, I'm surprised that there is not wider consensus on how to code an intrabar strategy backtest. Maybe I just don't understand how to properly use the lower time frame secondary series (i.e., 1 min) to create sound backtest results.

How are you backtesting your strategies? How would you backtest the simple MACross strategy? (Side note, clearly I'm not interested in trading the MACross, I'm just trying to use it to understand how to create a sound backtest).

Thanks for your help!

Aventeren

[aventeren]

EMA calls with DataSeries?

Just to confirm, if I my triggerDataSeries contains historical values equal to the corresponding bar indexed Close values (i.e., I Set triggerDataSeries[0] to Close at the end of OnBarUpdate) and a triggerDataSeries[0] value equal to a temporary "test" value, when I pass the triggerDataSeries to the NT EMA indicator, the value I receive back should be equal to what the EMA would be with the "test" value (i.e., triggerDataSeries[0] was Set to "test" value) and the historical Close values, correct?

IOW, the EMA doesn't care what DataSeries you give it, because it will calculate an EMA based on those values--and therefore the EMA can be passed a DataSeries type and does not require an IDataSeries type?

I'm seeing that there is a distinct difference between a DataSeries type (which just holds one bar indexed value) and an IDataSeries type (which appears to hold the bar indexed OHLC values). And furthermore, that many of the stock NT indicators require IDataSeries type inputs, which DataSeries are decidedly not?

Can someone please conclusively state that an IDataSeries type is NOT a DataSeries type? It just seems to me that they share the words "DataSeries"--but that their only common feature is that they are bar indexed, and that a DataSeries type will not work as passed parameter to an indicator that requires an IDataSeries type?

Boy we're in the weeds. I hope this isn't getting confusing. Am I overthinking this?

Thanks,

Aventeren