managed rule for historic Stop price is incorrect

[DaveE]

I am talking about a typical strategy using Calculate.OnBarClose and running in Historical.
Example scenario:
Strategy is Long, wants to set a Stop below the latest price.
Closing price before session break (Close[0]) was 103.815
Open of bar-being-built (the latest price at OnBarUpdate() is 104.02.
If this was realtime strategy should be able to select any price below 104.02 for a stop.
Lets say strategy detected the gap-up and wants to lock-in the gap-up by setting a stop at the last Close.
Problem: The current 'managed' rule gets in the way and incorrectly prevents this (even though if we were realtime the broker would allow it).

To reproduce the problem run the attached script (see settings recording in first line of output).

Example output:

DataLoaded: sTestExitLongStopLimit. Dataseries: Instrument=USDJPY, MarketDataType=Last, BarsPeriodType=Day, Value=1, Timeframe: From=01/09/2016 00:00:00, To=10/10/2016 00:00:00, TradingHours=Forex
Historical
CurrentBar=0 01/09/2016 22:00:00. Flat. EnterLong().
01/09/2016 22:00:00 Strategy 'sTestExitLongStopLimit/-1': Entered internal SubmitOrderManaged() method at 01/09/2016 22:00:00: BarsInProgress=0 Action=Buy OrderType=Market Quantity=100,000 LimitPrice=0 StopPrice=0 SignalName='' FromEntrySignal=''
OnOrderUpdate Name=Buy, OrderState=Submitted
OnOrderUpdate Name=Buy, OrderState=Accepted
OnOrderUpdate Name=Buy, OrderState=Working
OnOrderUpdate Name=Buy, Qty=100000, filled=100000, avgFillPrice=103.24
CurrentBar=1 02/09/2016 22:00:00. Long. New bar has opened higher at 104.02 than Close[0]=103.815, so SHOULD be OK to use ExitLongStopLimit(limitPrice=103.815, stopPrice=103.815)
02/09/2016 22:00:00 Strategy 'sTestExitLongStopLimit/-1': Entered internal SubmitOrderManaged() method at 02/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopLimit Quantity=0 LimitPrice=103.81'5 StopPrice=103.81'5 SignalName='' FromEntrySignal=''
Strategy 'sTestExitLongStopLimit/-1': A Sell stop order placed at '02/09/2016 22:00:00' has been ignored since the stop price is greater than or equal to the close price of the current bar. This is an invalid order and subsequent orders may also be ignored. Please fix your strategy.
02/09/2016 22:00:00 Strategy 'sTestExitLongStopLimit/-1': Ignored SubmitOrderManaged() method at 02/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopLimit Quantity=0 LimitPrice=103.81'5 StopPrice=103.81'5 SignalName='Sell' FromEntrySignal='' Reason='Invalid order price, please see log tab'
CurrentBar=3 06/09/2016 22:00:00. Long. New bar has opened higher at 102.035 than Close[0]=102.011, so SHOULD be OK to use ExitLongStopLimit(limitPrice=102.011, stopPrice=102.011)
06/09/2016 22:00:00 Strategy 'sTestExitLongStopLimit/-1': Entered internal SubmitOrderManaged() method at 06/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopLimit Quantity=0 LimitPrice=102.01'1 StopPrice=102.01'1 SignalName='' FromEntrySignal=''
06/09/2016 22:00:00 Strategy 'sTestExitLongStopLimit/-1': Ignored SubmitOrderManaged() method at 06/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopLimit Quantity=0 LimitPrice=102.01'1 StopPrice=102.01'1 SignalName='Sell' FromEntrySignal='' Reason='Invalid order price, please see log tab'
CurrentBar=9 14/09/2016 22:00:00. Long. New bar has opened higher at 102.41 than Close[0]=102.407, so SHOULD be OK to use ExitLongStopLimit(limitPrice=102.407, stopPrice=102.407)
14/09/2016 22:00:00 Strategy 'sTestExitLongStopLimit/-1': Entered internal SubmitOrderManaged() method at 14/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopLimit Quantity=0 LimitPrice=102.40'7 StopPrice=102.40'7 SignalName='' FromEntrySignal=''
14/09/2016 22:00:00 Strategy 'sTestExitLongStopLimit/-1': Ignored SubmitOrderManaged() method at 14/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopLimit Quantity=0 LimitPrice=102.40'7 StopPrice=102.40'7 SignalName='Sell' FromEntrySignal='' Reason='Invalid order price, please see log tab'

I highlighted the first two errors in red.

Lets call the current rule

A Sell stop order placed at DateTime has been ignored since the stop price is greater than or equal to the close price of the current bar.

A few points:
1) I dont see this rule documented anywhere. All the 'managed' rules that stop users doing things should be documented. And ideally each should be assigned a unique reference number. This would make discussion about these rules much easier, and should help NTs testing department.
2) The current rule logs its violations inconsistently: It only logs the first violation. But will show all of them in Output only if TraceOrders=true.
3) The rule SHOULD be

Sell stop order price has to be below Open of bar-being-built

(or if NT has Bid information) "Sell stop order price has to be below Open (Bid) of bar-being-built".

obviously something similar applies to Buy stop orders. Correct rule would be

Buy stop order price has to be above Open (Ask) of bar-being-built

And please I don't want to hear "this is expected behavior" or "NT has worked like this since version 1".
How could NT have been using an incorrect rule for so long?
1) Many users are using intra-day dataseries. So usually Close[0] will happen to be the same as Open of bar-being built. Difference may appear only after session break, where many people have learned not to trade.
2) Many users may be hitting this violation regularly, but ignoring it since only the first instance gets logged. So their problem may be much worse than log indicates and they might be getting a lot stops that should be valid silently 'ignored' thus causing unexpected order behavior in Historic.

[DaveE]

problem applies to Unmanaged also

I changed the title of this thread to remove 'managed' since I just noticed the problem applies to Unmanaged also.
Attached is sample script that shows same problem in IsUnmanaged = true

Extract from output:

Historical
* CurrentBar=0 01/09/2016 22:00:00. Flat. Entering Long.
01/09/2016 22:00:00 Strategy 'sTestHistoricStopUnmanaged/-1': Entered internal SubmitOrderUnmanaged() method at 01/09/2016 22:00:00: BarsInProgress=0 Action=Buy OrderType=Market Quantity=100,000 LimitPrice=0 StopPrice=0 SignalName=''
OnOrderUpdate Name=Buy, OrderState=Submitted
OnOrderUpdate Name=Buy, OrderState=Accepted
OnOrderUpdate Name=Buy, OrderState=Working
OnOrderUpdate Name=Buy, Qty=100000, filled=100000, avgFillPrice=103.24
* CurrentBar=1 02/09/2016 22:00:00. Long. New bar has opened higher at 104.02 than Close[0]=103.815, so SHOULD be OK to set stop at 103.815
02/09/2016 22:00:00 Strategy 'sTestHistoricStopUnmanaged/-1': Entered internal SubmitOrderUnmanaged() method at 02/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopMarket Quantity=100,000 LimitPrice=103.81'5 StopPrice=103.81'5 SignalName='SellStop'
Strategy 'sTestHistoricStopUnmanaged/-1': A Sell stop order placed at '02/09/2016 22:00:00' has been ignored since the stop price is greater than or equal to the close price of the current bar. This is an invalid order and subsequent orders may also be ignored. Please fix your strategy.
02/09/2016 22:00:00 Strategy 'sTestHistoricStopUnmanaged/-1': Ignored SubmitOrderUnmanaged() method at 02/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopMarket Quantity=100,000 LimitPrice=103.81'5 StopPrice=103.81'5 SignalName='SellStop' FromEntrySignal='' Reason='Invalid order price, please see log tab'
* CurrentBar=3 06/09/2016 22:00:00. Long. New bar has opened higher at 102.035 than Close[0]=102.011, so SHOULD be OK to set stop at 102.011
06/09/2016 22:00:00 Strategy 'sTestHistoricStopUnmanaged/-1': Entered internal SubmitOrderUnmanaged() method at 06/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopMarket Quantity=100,000 LimitPrice=102.01'1 StopPrice=102.01'1 SignalName='SellStop'
06/09/2016 22:00:00 Strategy 'sTestHistoricStopUnmanaged/-1': Ignored SubmitOrderUnmanaged() method at 06/09/2016 22:00:00: BarsInProgress=0 Action=Sell OrderType=StopMarket Quantity=100,000 LimitPrice=102.01'1 StopPrice=102.01'1 SignalName='SellStop' FromEntrySignal='' Reason='Invalid order price, please see log tab'

[koganam]

I am talking about a typical strategy using Calculate.OnBarClose and running in Historical.
Example scenario:
Strategy is Long, wants to set a Stop below the latest price.
Closing price before session break (Close[0]) was 103.815
Open of bar-being-built (the latest price at OnBarUpdate() is 104.02.
If this was realtime strategy should be able to select any price below 104.02 for a stop.
Lets say strategy detected the gap-up and wants to lock-in the gap-up by setting a stop at the last Close.
Problem: The current 'managed' rule gets in the way and incorrectly prevents this (even though if we were realtime the broker would allow it).

To reproduce the problem run the attached script (see settings recording in first line of output).

Example output:

I highlighted the first two errors in red.

Lets call the current rule
A few points:
1) I dont see this rule documented anywhere. All the 'managed' rules that stop users doing things should be documented. And ideally each should be assigned a unique reference number. This would make discussion about these rules much easier, and should help NTs testing department.
2) The current rule logs its violations inconsistently: It only logs the first violation. But will show all of them in Output only if TraceOrders=true.
3) The rule SHOULD be
(or if NT has Bid information) "Sell stop order price has to be below Open (Bid) of bar-being-built".

obviously something similar applies to Buy stop orders. Correct rule would be


And please I don't want to hear "this is expected behavior" or "NT has worked like this since version 1".
How could NT have been using an incorrect rule for so long?
1) Many users are using intra-day dataseries. So usually Close[0] will happen to be the same as Open of bar-being built. Difference may appear only after session break, where many people have learned not to trade.
2) Many users may be hitting this violation regularly, but ignoring it since only the first instance gets logged. So their problem may be much worse than log indicates and they might be getting a lot stops that should be valid silently 'ignored' thus causing unexpected order behavior in Historic.

You have this:

... case MarketPosition.Long:
//If the bar has gapped up then set ExitLongStopLimit using price from Close[0]
double openNew = Bars.GetOpen(CurrentBar+1);
if (openNew<=Close[0]) return;//price did not gap up

double stopPrice = Close[0];//this SHOULD be allowed since current price is higher
Print(string.Format("CurrentBar={0} {1}. Long. New bar has opened higher at {2} than Close[0]={3}, so SHOULD be OK to use ExitLongStopLimit(limitPrice={3}, stopPrice={3})",CurrentBar, Time[0], openNew, stopPrice));
ExitLongStopLimit(stopPrice, stopPrice);
break;

One cannot peek into the future with CurrentBar+1. In NinjaTrader, bars should be evaluated with the distinct restriction and viewpoint that the last valid bar is the CurrentBar, and the code must be written that way. That means that one must anchor oneself in the present and look at the past, especially on bars that will be processed based only on the OHLC values, as are any actual or quasi-historical bars.

...				case MarketPosition.Long:
					//If the bar has gapped up then set ExitLongStopLimit using price from Close[0]
					double openNew = Bars.GetOpen(CurrentBar);
					if (openNew<=Close[1]) return;//price did not gap up
					
					double stopPrice = Close[1];//this SHOULD be allowed since current price is higher
					Print(string.Format("CurrentBar={0} {1}. Long. New bar has opened higher at {2} than Close[0]={3}, so SHOULD be OK to use ExitLongStopLimit(limitPrice={3}, stopPrice={3})",CurrentBar, Time[0], openNew, stopPrice));
					ExitLongStopLimit(stopPrice, stopPrice);
					break;

Simpler yet, just use Close[0] instead of Bars.GetOpen(...).

[NinjaTrader_PatrickH]

Thank you for the suggestions, DaveE.

I will forward these over to development.

[DaveE]

Hello koganam,

One cannot peek into the future with CurrentBar+1. In NinjaTrader, bars should be evaluated with the distinct restriction and viewpoint that the last valid bar is the CurrentBar

But this code

 openNew = Bars.GetOpen(CurrentBar+1);

is not 'peeking into the future' it is the latest price that is available at the point that OnBarUpdate() is called.

one must anchor oneself in the present and look at the past

I completely agree. But that is precisely what this code is doing. OnBarUpdate() is called after receiving the first tick of a new bar. That tick has the price that is going into openNew so it is (milliseconds) in the past. Whereas Close[0] might be days in the past (after a weekend).

Simpler yet, just use Close[0]

But my whole point is that Close[0] can be way out of date (especially after session break). Please look at the scenario in post#1.

[koganam]

Hello koganam,



But this code

 openNew = Bars.GetOpen(CurrentBar+1);

is not 'peeking into the future' it is the latest price that is available at the point that OnBarUpdate() is called.

I completely agree. But that is precisely what this code is doing. OnBarUpdate() is called after receiving the first tick of a new bar. That tick has the price that is going into openNew so it is (milliseconds) in the past. Whereas Close[0] might be days in the past (after a weekend).


But my whole point is that Close[0] can be way out of date (especially after session break). Please look at the scenario in post#1.

I understand what you are saying because I have been down that road. No, CurrentBar is the bar of the most recent tick. In any context, that means that there can be no bar more recent than CurrentBar. That is the latest price that is available at the point that OnBarUpdate() is called.

Print out the values and you will see that NT is under the covers reading that as Math.Min(CurrentBar, CurrentBar + 1//or whatever you put here), so you get no error on trying to access the future bar. It is still impossible in a logical sense, but NT seems to allow it in the mathematical sense because there is a hidden validator.

[DaveE]

CurrentBar is the bar of the most recent tick

If you are using Calculate.OnBarClose then when you are in OnBarUpdate():
CurrentBar is actually referring to the latest completed bar.
NT has already started a new bar but it is dangerous to use just about anything in it for normal use since it mainly refers to future stuff that has not been finalized. This is presumably why NT keeps this bar-being-built hidden away. But there is at least one useful piece of information and that is the Open price.

Print out the values

Thats what the strategy does! Try running the strategy on some historical daily bars. Compare the output with the chart. You will see that openNew is giving you valid information. Normally Open[0] is giving you a price that is one bar out of date, here you get it when it arrives.

[koganam]

If you are using Calculate.OnBarClose then when you are in OnBarUpdate():
CurrentBar is actually referring to the latest completed bar.
NT has already started a new bar but it is dangerous to use just about anything in it for normal use since it mainly refers to future stuff that has not been finalized. This is presumably why NT keeps this bar-being-built hidden away. But there is at least one useful piece of information and that is the Open price.


Thats what the strategy does! Try running the strategy on some historical daily bars. Compare the output with the chart. You will see that openNew is giving you valid information. Normally Open[0] is giving you a price that is one bar out of date, here you get it when it arrives.

Even granted that, it is evident that the Order Management Engine is built with the intention that validation is done on the last bar as supported by the platform. In other words it might be possible to access price data in the manner that you say, but they evidently do not think that that is supported use.

I would love it if things worked the way that you wrote, as my often cumbersome workarounds would not be necessary. However, such is the nature of using features that a vendor does not support. I am leery that they are willing to change the whole framework just to help us out here.

But, who knows, maybe I am just a jaded, cynical curmudgeon? We shall just have to see what comes our way from this.

[NinjaTrader_PatrickH]

Thank you for the suggestions, DaveE.

I will forward these over to development.

This suggestion has been assigned id SFT-1216.

[DaveE]

Here's a quote from Ray (CEO) which backs up what I am saying

Signals are generated on the close of a bar thus the next tick (which is the open of the next bar) is where the order is filled at. Limit and stop orders are "working" on the next bar and evaluated against this bars data to determine if it indeed has been filled and at what price.

What Ray here calls "next bar" is what I am calling bar-being-built.
Unfortunately whoever wrote and implemented the rule for checking price of historical Stop orders does not seem to have been listening to the CEO.

[DaveE]

This suggestion has been assigned id SFT-1216

If this change is made it should probably be done in the same release as the suggested fix for Historical behavior of GetCurrentBid() & GetCurrentAsk(). This is because user strategies often use GetCurrentBid() minus some ticks to assign a price for a stop order on a Long (for example), and both GetCurrentBid() and the Historical stop order rule are currently looking at the same (potentially days out-of-date) price.

[NinjaTrader_PatrickH]

Here's a quote from Ray (CEO) which backs up what I am saying

What Ray here calls "next bar" is what I am calling bar-being-built.
Unfortunately whoever wrote and implemented the rule for checking price of historical Stop orders does not seem to have been listening to the CEO.

Please refer to my post at the following link that explains why this is not the same thing: http://ninjatrader.com/support/forum...70&postcount=3

[DaveE]

Please refer to my post at the following link that explains why this is not the same thing: http://ninjatrader.com/support/forum...70&postcount=3

I get 'invalid link'

[NinjaTrader_PatrickH]

Hello DaveE,

Thank you for your response.

My post was removed in favor of the direction my colleague took at the following link: http://ninjatrader.com/support/forum...ad.php?t=91168