Orders-update callbacks does not reflect the natural order of events as they happened

[misova]

I found strange behavior of Ninjatrader 7 API:
Problem is, that order-state is updated faster, than allows required logic of execution callbacks.

Follow this example:

Entry order 2 cts is filled in 2 quick executions of 1 cts.

One would expect this order of callbacks:

1. Execution 1 cts
2. Execution 1 cts
3. Order update - new status Filled

But in reality, NinjaTrader randomly (it just happens this way sometimes) invokes callbacks in this order:

1. Execution 1cts
2. Order update - new status Filled
3. Execution 1 cts

We all know, this can never happen in reality - i.e. Order state cannot get into state Filled, before both executions are registered = these execution callbacks should be first invoked by Ninja API, and then - Order-update callback with new order-state Filled should be invoked.

What it means ?
It means, that 2 executions were very fast one after another and NinjaTrader immediately updates order-state to most up-to-date order-state (in this case Filled). It does not care, if proper sequence of execution-callbacks (that had to happen before) were called first.

That means - strategy can be informed about order update to state Filled, sooner, than it is informed about executions of the same order. This does not make sense- because it is vice-versa. API should order this events into real and correct order of the events.

Is this correct behaviour? Is it OK, that order of callbacks does reflect the order of real events as they happened?
Should not be guaranteed, the all execution callbacks are invoked first, before the order-update-callback will be called with status "Filled"?

In case this is not valid behavior - it is important, that this does not happen in new NT8 too.

[NinjaTrader_PatrickH]

Hello misova,

Thank you for your post.

What API/brokerage technology are you using when this is seen?

[misova]

Hi,
this happens in our case with Interactive Brokers (live account, not demo account)

[koganam]

Hello misova,

Thank you for your post.

What API/brokerage technology are you using when this is seen?

For me, the same thing happens on Market Replay, especially so when I use an smaller time-frame (1-second) to enter on a signal triggered by the close of the bar on the higher time frame. (COBC = true).

[NinjaTrader_PatrickH]

Hello misova,

Thank you for your patience.

I did overlook the fact that OnExecution would be called after OnOrderUpdate. So in a 2 quantity order that filled with two executions we would see the sequence of OnExecution, OnOrderUpdate, and then OnExecution again. This is mentioned under the OnExecution section of the Help Guide: http://ninjatrader.com/support/helpG...nexecution.htm

This would be the expected sequence of events in NinjaTrader 8 as well.

[misova]

Thank you Patrick for information.
I see the text, you mentioned in help:

OnExecution is always called after OnOrderUpdate() is called

but can't understand this logic behind it. In case we agree, that real order of events happens this way:

1. First order-execution
2. then order-state update to PartFilled / Filled

then what's the point NinjaTrader API switches this order vice versa - opposite as they really happened?

It does not make sense to me right now and makes code of our strategies more complex,
to handle this logic of NT API.

Could you explain please, what's the motivation behind this behavior? How does it make sense or why it is implemented this way?

[NinjaTrader_Brett]

Its the other way around technically. An execution is another term of saying a part filled / filled order update came in.

Basically the order event comes in from the exchange as saying hey your order filled. We mark the order as filled and since we marked an order as filled we then generate an Execution event for it.

Let me know if any questions.

[misova]

Hi Brett,
thank you for explanation.

I can understand, what you mean. But let me explain:
I think, that reversed order of events is not a minor nuance , but invalid and inconsistent behavior. Let me explain more:

Currently, Ninja informs us - "Hey, you order was completely filled". But if we try to find consistent information about executions, that lead to thisFilled order, we cannot get it. We cannot get it, despite it is known and executions already exist. This information about executions is already available (because executions already happened before order-Filled).

This forces us developers to compensate this reversed order in our algos.
We have to wait for future execution-callback, to acquire the missing part of whole picture = to get executions, that lead to the previously reported order-Filled.

I suggest, that this behavior needs to be fixed = reversed and reflect real order of events.
Current behavior makes confusion, because Ninja

1. first notifies about event, that happened later = filling order happened later
2. later notifies previous events - executions (executions happened before order-Filled, but are reported later)

If we accept, that:

• The order of events is reversed
• It does not reflect reality.
• It forces us developers to compensate and handle this reversed order of events in our algos.

So why not to fix it - at least in NinjaTrader 8 ?
I think, correct order of reported events is critical issue of any trading API and needs to work correctly in order, how events really happen. Reversed order of events causes problems and brings new level of complexity, that we need to understand first, and after that compensate this strange event-order in our algos.

It should be fixed and made correct. - which results in clean, consistent and primarily correct trading API.

That's my suggestion - let's create an issue for this and fix this at least in NT8.

[NinjaTrader_Brett]

Sorry not sure we are on the same page and I'm trying to understand why you might be thrown off. The expectation is OnOrderUpdate then OnExecution. We only know to generate an on Execution event after a filled order has been processed.

Technically you don't even need OnExecution. You can forget about it, its a convenient event to handle purely order fills. You could code all your logic in OnOrderUpdate and if Order == Filled || PartFilled then do your logic.

Hopefully that will help demonstrate why it is how it is. There are no plans to change the sequence of events we have been using for X years and find the sequence to be intuitive and correct and do not understand why you feel its backwards.

-Brett

[misova]

Ok, I will try to simplify it...

Can we agree on one thing?
that Execution-event happens first and based on that = later, order-state is updated ? Can we agree that it markets work in this order and not vice versa?

Example:
Let's have Limit order of size 10 cts.
Market touches this order, and one small execution (3 cts) happens, which results, that 3 of 10 contracts of this order are filled. So we have execution first. Based on this execution -> order-state is updated to "PartFilled".

After 2 minutes, market touches the original order again, new execution with size 7cts happens. We have execution(s) again. And execution(s) influences the order-state.
Based on these additional execution(s), order-state value is updated to "Filled."

As we see, executions happen first, and they influence order-state PartFilled | Filled.

All I want to demonstrate by the example, is that :
executions are first. Based on the executions i.e. "after these executions"- order-state gets updated to state "PartFilled | Filled".

The same said vice-versa: I wanted to demonstrate, that executions do not come later = after order-state changes to "PartFilled | Filled". Executions happen first, and these execution cause, that order-state gets updated to "PartFilled | Filled"


Can we agree, that order of real events is that first executions, and after that order-state changes to Filled | Partfilled?

If yes, than all what I want is:
NinjaTrader should report these events in real and correct order

It is so simple. API should reflect real order of events. That's all I want to report.

Reason, why I report it:
We have a complex framework built on top of the NT API framework and we found out this inconsistency, which causes problem for us. But our framework is not important, we can change it internally.
Important is, if NT API works correctly and we can rely on it. I think I found invalid behavior, so I would like to understand it or fix it.

I think that reasoning like "behavior is mentioned in help" or "it used that way a long time", does not make sure, it is correct.

[NinjaTrader_Brett]

No we cannot agree on that fact and I think that's the source of the confusion.

Technically what starts the whole chain off is the API reporting an order fill.

[misova]

Yes, I agree - we found a point of confusion.

So am I true, if I say, that? :
No executions are reported from exchange - only order-updates are reported from exchange. And Ninja reconstructs these order-update events, to create artificial execution events, which are reported later?

[NinjaTrader_Brett]

Cant comment on exchanges since we do not work directly with exchanges but work with API's such as CQG, Continuum, Rithmic etc. In the majority of cases that statement is true meaning we just get an order filled event and NT then generates an Execution event after receipt.

[misova]

Uff, that's completely new for me. I would never say that.

Thank you Brett for all your patience.
I appreciate it, that we came to this conclusion.

Need some time to absorb this news.

[koganam]

Yes, I agree - we found a point of confusion.

So am I true, if I say, that? :
No executions are reported from exchange - only order-updates are reported from exchange. And Ninja reconstructs these order-update events, to create artificial execution events, which are reported later?

Pretty much. Just as he has already said.

Regardless, the execution is simply a report that a trade filled. To draw a macabre parallel, in places where they are still beheading prisoners, the prisoner could only be reported as executed, after it was clear that the axe had severed his head. First came the severance, then the report of the execution, which is a confirmation that the prisoner's head was severed.

Similarly, the Execution event is a report that an order was successfully executed, which means that it must first be confirmed that the order filled. The only real reason to use OnExecution() is that it is, in of itself, a confirmation that NT says that it has successfully received notice of the order filling.