Correct Way to Close PartFilled Order in AddOn

[jeronymite]

Scenario: An AddOn submits an Order. The Order becomes PartFilled. The AddOn needs to close the PartFilled Order.

What is the "correct" way to do this?

My approach has been:

1. Cancel the Order [intending thereby to cancel the remaining unfilled quantity]
2. Submit an appropriate Sell/Buy-to-Cover Order to close out the already filled quantity

Advice on this much appreciated.

Thanks.

[bltdavid]

Scenario: An AddOn submits an Order. The Order becomes PartFilled. The AddOn needs to close the PartFilled Order.

What is the "correct" way to do this?

My approach has been:

1. Cancel the Order [intending thereby to cancel the remaining unfilled quantity]
2. Submit an appropriate Sell/Buy-to-Cover Order to close out the already filled quantity

Advice on this much appreciated.

Thanks.

Your approach sounds correct to me.
It's really a matter of how many situations are you wanting to be
prepared to handle.

If this is an entry order, make sure you also cancel any stop and/or
target orders. I mean, if an entry order PartFilled, there may already
be active stop and target orders for the partial filled quantity.

Also, if this is an entry order, you may want to check how
NinjaTrader 7 handled the chart trader Close button. Those
considerations may still be worthwhile. (Sorry, not sure where
in the NT8 docs that same info appears).

Finally, make sure you watch out for overfills. I mean, think about
that carefully. The order PartFilled, so market price may still be
lingering near the filled price, which means market price may
bounce around enough to cause the entry order to fill while your
cancel order is in transit, which causes the overfill situation. What
to do with an overfill? Well, sounds like you'll want to close the
position that was opened/augmented by the overfill -- and, just to
be clear, since this is an AddOn, you may need to write the overfill
detection logic yourself.

-=o=-

As far as "close the PartFilled order", this is worded incorrectly.
I mean, orders aren't technically 'closed', which I'm sure you know.
You cancel orders, but you close (or exit) positions.

Sorry, not being condescending. I bring this up to point out that the
action verbs used by NinjaScript APIs can help -- you'll note that the API
does not have a 'CloseOrder' or 'CancelPosition/Long/Short' methods,
NT methods tend to use 'Cancel' for orders and 'Entry' and 'Exit' for
positions -- so the correct English wording helps to reinforce the API,
and vice versa -- which can help to avoid confusion.

[jeronymite]

Much appreciated bltdavid . Yes, I was speaking a bit loosely with the terminology -- I'll tighten that up. Good point.

I'd wondered about overfill, and although I have logic to recognise those situations, I'll revisit it to be sure it handles it well.

I debated whether one should cancel then exit or vice versa, and I decided that doing the cancel first reduces the likelihood of overfills by reducing the time available for more partial fills.

I presume the final OrderState would still be PartFilled, even if it is also "PartCanceled". It would seem unlikely that it would change to Filled after the Cancel ... but strange things have been known to happen.

Thanks.

[bltdavid]

Much appreciated bltdavid . Yes, I was speaking a bit loosely with the terminology -- I'll tighten that up. Good point.

I'd wondered about overfill, and although I have logic to recognise those situations, I'll revisit it to be sure it handles it well.

I debated whether one should cancel then exit or vice versa, and I decided that doing the cancel first reduces the likelihood of overfills by reducing the time available for more partial fills.

I presume the final OrderState would still be PartFilled, even if it is also "PartCanceled". It would seem unlikely that it would change to Filled after the Cancel ... but strange things have been known to happen.

Thanks.

That's not quite right. There is no OrderState known as 'PartCancelled'.
[EDIT: Doh, my bad, that's probably why you put it in quotes -- apologies.]

When you cancel an order, the normal path is through these OrderStates:

• 'CancelPending' (which should be very quick, probably just means CancelOrder has been called) and then,
• 'CancelSubmitted' (NinjaTrader waits for broker/exchange to respond, this is done asynchronously) and then,
• 'Cancelled' (asynchronous confirmation rec'd that order is cancelled at broker/exchange) which means,

the order has reached an absolute final state, known as a terminal state.

-=o=-

This normal state change can be interrupted precisely because the order
unexpectedly gets filled (or further partial filled) at the exchange just before
the exchange actually confirms the cancellation.

That is, it should be possible for states 'CancelPending' and 'CancelSubmitted'
to move to states 'Filled' or 'PartFilled'. If that happens, you have an overfill.

So, even though an overfill has occurred, the order is not necessarily done.
It must still continue its journey through the broker and/or exchange until its
OrderState has reached a terminal state.

Let's consider the remainder of that journey.
Remember, the overfill has occurred, but what happens next to the order?

Just like 'Cancelled', the states 'Filled' and 'PartFilled' are received via an
asynchronous socket, which means it could happen at any moment.

What I mean is, NinjaTrader is always listening for exchange messages,
and if a Filled or PartFilled msg arrives for an order in states CancelPending
or CancelSubmitted, then Filled means the order has reached its absolute
final terminal state -- meaning, its state will not change again.

Whereas, if a PartFilled msg arrives, the order can still move to Filled or
Cancelled, depending upon what happens next at the exchange (either
the auction engine runs first and fills more of the order, or the cancellation
takes effect and the order is removed from the auction).

My point is: once an order has reached a terminal state, such as Filled or
Cancelled, the OrderState of that order is final and it will not change again.

And the corollary is this:
Until the OrderState is in a terminate state, the order ain't done.

-=o=-

Not to rag on you, but realize "Filled after the Cancel" is imprecise.

Why? Because (see above) cancel is really broken into 3 different state changes.

Getting filled after 2 of the 3 cancel states is very much possible (and this is
where overfills come from), but you will never get filled after the 3rd and final
cancel OrderState known as 'Cancelled'.

[bltdavid]

Just to be thorough, I'll add a few more notes.

An overfill is not an OrderState.
There is no OrderState value that indicates an overfill has occurred.

An overfill is not an ErrorCode.
There is no ErrorCode value that indicates an overfill has occurred.

An overfill is not a Rejected order.
Far from it, an overfill means the order filled or partial filled -- that's
not a rejection at all -- the exchange found the order very acceptable,
and performed work to fill it.

An overfill is a condition. It is the byproduct of normal and correct
events happening to an order which is -- for us -- an inconvenient
possibility.

An overfill is usually always about the failure to fully cancel an order.
If a pair of OCO entry orders bracketing market price both happen to
fill (How does that happen? When price is extremely volatile, after
the first order fills, price reverses and fills the second order before
the second order is cancelled via OCO), that is also called an 'overfill'.

A overfill is like a race condition, the timing of events is everything.

-=o=-

Overfills may be treated like errors, which says something about the
level of sophistication handling an overfill -- but that doesn't really
mean the overfill is the error -- the error is more likely in the logic
of the code handling the orders.

Remember, an overfill is a condition. It's a manifestation of events.
If you see a lot of overfills, you're probably doing something wrong
in your code.

-=o=-

A good way to read more about overfills is using this search.

[jeronymite]

Thanks again for your insights, bltdavid . I understand the nature of terminal state and the various paths things take to end up there, but it is good that you point out what you have. It also addresses my musing on the final state of a PartFilled and subsequently cancelled order. And the possible ways an overfill situation may arise are also understood. Again, always good to be reminded.

I think one of the most important things to remember about overfills (and a variety of other possibilities!) is that we are dealing with real-time systems when we trade, and both within our own code (i.e. independent of external events) as well as (particularly) external events, the need to be very mindful of the possibility for race conditions is extremely important. Overfills are a great example, and again, thanks for using that terminology to highlight the fact.

One general comment: it is always good to be reminded about how things really work -- whether one knows these things or not, a gentle reminder can sometimes jog a memory or cause a neuron to fire in a way it had previously bypassed for some reason. I appreciate your contribution to keeping one's thinking accurate and sharply focused.

Thanks.

[NinjaTrader_ChelseaB]

Hello jeronymite,

bltdavid is right on the money as always.

Just one additional note I did not see mentioned is that while an overfill would not cause the order to have an Rejected state, there would likely be an order error of 'Unable to cancel order'. With an addon, this would not stop the code as a strategy would, and can be detected in the <Account>.OrderUpdate event.

[bltdavid]

Just one additional note I did not see mentioned is that while an overfill would not cause the order to have an Rejected state, there would likely be an order error of 'Unable to cancel order'. With an addon, this would not stop the code as a strategy would, and can be detected in the <Account>.OrderUpdate event.

Yep, with all the details I was trying to dump, I forgot that one.

Good catch, thanks!

I agree. We should keep a watch out for ErrorCode value of
'UnableToCancelOrder' -- which should be a byproduct of the
failure of the exchange to cancel an already filled order.

In other words, if/when you ever see 'UnableToCancelOrder',
probably coupled with 'OrderState.Filled', the reason could be
due to an overfill condition.

Hmm ... come to think of it ...

That specific OrderState/ErrorCode combination sounds like a
sure fire way to detect an overfill -- but it's not guaranteed that
every overfill will manifest that exact combination.

Or is it?

Chelsea, your thoughts?

[NinjaTrader_ChelseaB]

bltdavid,

That would be a pretty good way to know it was an overfill, but note there can be other reasons for an 'unable to cancel order' that may not be an overfill, and the order just becomes filled in time (if there is volume for it). The error message itself (from most brokerages) will actually indicate its an overfill.

[bltdavid]

The error message itself (from most brokerages) will actually indicate its an overfill.

Good to know.

Thanks, Chelsea!

[jeronymite]

Thanks. I'm aware of the possibility of being unable to cancel and am already catching and handling that error, along with logic for overfills. Appreciate your thoughts, Chelsea.

Thanks.