Order of Order Execution when using Historical Data

[StefanJ]

Hello,

I am using a managed approach to submit orders.
My strategy throws an exception when a position is not closed by the expected stop-loss or take-profit order.

At bar 6240 order "lG.18" was submitted (buy on stop).
At bar 6245 the strategy opened a new short position via order "sG.15" and submitted the bracket orders "sG.15.sl" and "sG.15.tp" in OnExecutionUpdate.
At each following bar, the stop price for the orders are adjusted.

According to the log output, the orders are changed at bar 6260:

1. Order "sG.15.sl", BuyToCover, StopMarket at 56.53
2. Order "lG18", Buy, StopMarket at 56.53.

My assumption is that the order of execution of orders with same direction (BuyToCover, Buy) and the same StopPrice is determined by the point of time where the orders were submitted or modified. In this case, order "lG18" was modified after "sG.15.sl" so that order "sG.15.sl" would be executed first.

But in fact, order "lG18" was executed before order "sG.15.sl" (this happens when using historical data, strategy was not used with realtime data yet).
Finally, such a scenario happens multiple times (e.g., when using different instruments or time frames) but this is the only case where the orders are not executed in the expected order.

So my question is:

• Is my assumption regarding the order of order execution correct?
• Is this only an issue with historical data?

Thanks!

[SIZE=8px][FONT=courier new][6260] OnBarUpdate Time=02/20/2019 15:26:00 -> Close=56.44
 OnOrderUpdate NT-00095-34: Name=sG.15.sl; limit=0.0; StopPrice=56.53; Qty=1; Filled=0; State=ChangeSubmitted; Time=15:00:02; Err=NoError
 OnOrderUpdate NT-00095-34: Name=sG.15.sl; limit=0.0; StopPrice=56.53; Qty=1; Filled=0; State=Accepted; Time=15:00:02; Err=NoError
 OnOrderUpdate NT-00095-34: Name=sG.15.sl; limit=0.0; StopPrice=56.53; Qty=1; Filled=0; State=Working; Time=15:00:02; Err=NoError

 OnOrderUpdate NT-00094-34: Name=lG.18; limit=0.0; StopPrice=56.53; Qty=1; Filled=0; State=ChangeSubmitted; Time=15:00:02; Err=NoError
 OnOrderUpdate NT-00094-34: Name=lG.18; limit=0.0; StopPrice=56.53; Qty=1; Filled=0; State=Accepted; Time=15:00:02; Err=NoError
 OnOrderUpdate NT-00094-34: Name=lG.18; limit=0.0; StopPrice=56.53; Qty=1; Filled=0; State=Working; Time=15:00:02; Err=NoError

[6261] OnBarUpdate Time=02/20/2019 15:28:33 -> Close=56.44
 OnOrderUpdate NT-00094-34: Name=lG.18; limit=0.; StopPrice=56.5300; Qty=1; Filled=1; AvgP=56.53; State=Filled; Time=15:00:02; Err=NoError; Comment=

Strategy 'G201902S/-1': Error on calling 'OnOrderUpdate' method on bar 6261: 
  The names of opening and closing order do not match: 
  opening name=sG.15
  closing name=lG.18 but shall be sG.15.sl or sG.15.tp[/FONT][/SIZE]

[StefanJ]

There is a typo: "I am using a managed approach to submit orders" -> unmanaged approach

[NinjaTrader_ChelseaB]

Hello StefanJ,

Welcome to the NinjaTrader Forums!


The output isn't perfectly clear and the code was not provided but I am seeing the error at the bottom.

However, the error is coming from OnOrderUpdate() after the lG.18 order has filled?

What is the line of code in OnOrderUpdate() that is causing this error?


The order action should be determined by the position when the order is placed.

A Buy order is used to enter or increase a long position when the position is flat or long.
A BuyToCover order is used to exit a position when the position is short.
A Sell order is used to exit a position when the position is long.
A SellShort order is used to enter or increase a short position when the position is flat or short.

However, based on the error message being about the order name I'm not certain the order action is causing the error.


Is there a position opened, and then sG.15.sl is working, then while this is working lG.18 is placed and fills and the sG.15.sl remains working after the position has changed?

[StefanJ]

Hello Chelsea,

Thanks for your answer.

Today, I detected a similar problem with a different instrument (Brent Future, BZ-05-19) which make the issue clearer.

According to the log output, the following happened:

1. At bar [0087] order 'lX.2' was submitted (Buy StopMarket at 60.79)
2. At bar [0158] order 'lX.2' - Buy StopMarket price changed to 60.78
3. At bar [0161] order 'lX.2' was filled at 60.78
4. At bar [0161] after the fill, stop-loss order 'lX.2.sl' was submitted (Sell StopMarket at 60.61)
5. At bar [0162] order 'sX.1' was submitted (Sell StopMarket at 60.60)
6. At bar [0168] order 'sX.1' was filled at 60.60

In summary,
lX.2 -> (NT-00003-2655) -> Long at 60.78
lX.2.sl -> (NT-00004-2655) -> Sell StopMarket StopPrice=60.6100 in bar [0161]
sX.1 -> (NT-00006-2655) -> Sell StopMarket StopPrice=60.6000 in bar [0162]
==> Order 'lX.2.sl' should have been executed before order ('sX.1')


So, the strategy is Long BZ0519 at 60.79. Then a stop-loss order is submitted and pending (stop at 60.61). Later a go Short order was submitted (sell at 60.60 on stop market).

Exchanges use an order queue and order 'lX.2.sl' should have been executed (and reported to NinjaTrader) before order 'sX.1' because order 'lX.2.sl' uses a higher StopPrice than order 'sX.1'. (In addition, order 'lX.2.sl' was submitted before order 'sX.1'.)

I guess, that the NinjaTrader's simulation of order execution when using historical data does not use an order queue having the same behavior as the exchange's order queue.

Otherwise, I cannot see any reason why a sell order with a lower sell price is executed before a sell order with a higher sell price.

Best regards,
Stefan J.


State.Historical

[0087] Time=01/10/2019 07:22:30 -> Close=60.78
-- Submit Enter Long order lX.2: LONG on stop at 60.97 (close=60.78) ----
Submit order: SelectedBars=0; Action=Buy; Type=StopMarket; Qty=1; Limit=0; Stop=60.97; OCO=; Name=lX.2
... OnOrderUpdate NT-00003-2655: Name=lX.2; limit=0; StopPrice=60.97; Qty=1; Filled=0; AvgP=0; State=Submitted; Time=07:22:30; Err=NoError
... OnOrderUpdate NT-00003-2655: Name=lX.2; limit=0; StopPrice=60.97; Qty=1; Filled=0; AvgP=0; State=Working; Time=07:22:30; Err=NoError

[0158] Time=01/10/2019 09:59:54 -> Close=60.65
change order lX.2: LONG on stop at 60.78 (from 60.79)
... OnOrderUpdate NT-00003-2655: Name=lX.2; limit=0; StopPrice=60.78; Qty=1; Filled=0; AvgP=0; State=ChangeSubmitted; Time=07:22:30; Err=NoError
... OnOrderUpdate NT-00003-2655: Name=lX.2; limit=0; StopPrice=60.78; Qty=1; Filled=0; AvgP=0; State=Working; Time=07:22:30; Err=NoError

[0161] Time=01/10/2019 10:00:47 -> Close=60.74
... OnOrderUpdate NT-00003-2655: Name=lX.2; limit=0; StopPrice=60.78; Qty=1; Filled=1; AvgP=60.78; State=Filled; Time=07:22:30; Err=NoError
-- Submit bracket order
Submit order: SelectedBars=0; Action=Sell; Type=StopMarket; Qty=1; Limit=0; Stop=60.61; OCO=lX.2.oco.235618; Name=lX.2.sl
... OnOrderUpdate NT-00004-2655: Name=lX.2.sl; limit=0; StopPrice=60.61; Qty=1; Filled=0; AvgP=0; State=Submitted; Time=10:00:47; Err=NoError
... OnOrderUpdate NT-00004-2655: Name=lX.2.sl; limit=0; StopPrice=60.61; Qty=1; Filled=0; AvgP=0; State=Working; Time=10:00:47; Err=NoError
Submit order: SelectedBars=0; Action=Sell; Type=Limit; Qty=1; Limit=62.78; Stop=0; OCO=lX.2.oco.235618; Name=lX.2.tp
... OnOrderUpdate NT-00005-2655: Name=lX.2.tp; limit=62.78; StopPrice=0; Qty=1; Filled=0; AvgP=0; State=Submitted; Time=10:00:47; Err=NoError
... OnOrderUpdate NT-00005-2655: Name=lX.2.tp; limit=62.78; StopPrice=0; Qty=1; Filled=0; AvgP=0; State=Working; Time=10:00:47; Err=NoError
... OnPositionUpdate

[0162] Time=01/10/2019 10:02:32 -> Close=60.79
-- Submit Enter Short order sX.1: SHORT on stop at 60.60 (close=60.79) ----
Submit order: SelectedBars=0; Action=SellShort; Type=StopMarket; Qty=1; Limit=0; Stop=60.60; OCO=; Name=sX.1
... OnOrderUpdate NT-00006-2655: Name=sX.1; limit=0; StopPrice=60.60; Qty=1; Filled=0; AvgP=0; State=Submitted; Time=10:02:32; Err=NoError
... OnOrderUpdate NT-00006-2655: Name=sX.1; limit=0; StopPrice=60.60; Qty=1; Filled=0; AvgP=0; State=Working; Time=10:02:32; Err=NoError

[0168] Time=01/10/2019 10:10:59 -> Close=60.64
... OnOrderUpdate NT-00006-2655: Name=sX.1; limit=0; StopPrice=60.60; Qty=1; Filled=1; AvgP=60.6; State=Filled; Time=10:02:32; Err=NoError

Strategy G201902S/-1: Error on calling OnOrderUpdate method on bar 168:
The names of opening and closing order do not match:
opening=lX.2
closing=sX.1 but shall be lX.2.sl or lX.2.tp

lX.2 -> (NT-00003-2655) -> Long at 60.78
lX.2.sl -> (NT-00004-2655) -> Sell StopMarket StopPrice=60.61 in bar [0161]
sX.1 -> (NT-00006-2655) -> Sell StopMarket StopPrice=60.60 in bar [0162]

==> Order lX.2.sl should have been executed before order (sX.1)

[NinjaTrader_JoshG]

Hello StefanJ,

Chelsea is out of the office today, but will get back to you as soon as he returns.

Thanks for your patience.

[NinjaTrader_ChelseaB]

Hello StefanJ,

With SubmitOrderUnmanaged(), the signalName of the order is used on the chart and in the performance. It should be possible to send any order type at any time no matter what order they are in.
There are no special order handling rules like with the managed approach.

https://ninjatrader.com/support/helpGuides/nt8/en-us/submitorderunmanaged.htm

I also would not be expecting the order used to enter a position to have the same name as an order to close a position. It should be a different order with a different that exits the position.

Entering the market with Buy as the OrderAction and StopMarket as the OrderType, and then once that has filled and the OrderState is OrderState.Filled then submitting a new order after this point that is a Sell as the OrderAction and StopMarket as the OrderType to exit that position.

Please let me know if you would like a demonstration of this.

When it comes to historical backtesting, 1 tick intra-bar granularity is necessary for accuracy with order of events that are OnBarUpdate based, and for accuracy of fill prices.

Below is a link to a forum post on intra-bar granularity.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/94098-isfirsttickofbar-vs-onbarclose-for-backtest-live?p=773377#post773377

I'm finding any documention or other threads on this error message. Is this error message a native message coming from the brokerage? Are the return lines where the error is appearing on multiple lines from the message returning from the brokerage?