Inadvertently Going Short - Mitigation Options

[ChainsawDR]

While running a long only strategy, I've recently been having some infrequent issues whereby the strategy inadvertently goes short and shuts down (leaving the account position in a short state). It's proving tricky to debug, but I think I'm making progress (I have different rules for when a stoploss should be updated, and I was inadvertently sending more than one update when multiple rules' conditions were met - I'm now preventing multiple stoploss updates being sent on the same OnBarUpdate), but I'm looking for a backup strategy to resolve being Short in case it happens again as I suspect the issue may also be related to instances where the broker closes a position (because the stop price was met) but I'm sending a new stoploss order before the strategy realizes the position has already been closed.

If/When the first strategy becomes disabled, I'm thinking of running a second strategy (in addition to the primary strategy) that exits an account position if its detected as being Short. Aka if strategy is Flat & Account position is Short, exit the position. Am I correct in thinking that "Immediately submit, synchronize account" is the ideal synchronization setting for this purpose? Would this only sync at startup, or would it take the desired action if it detected the account position was Short at any time? Any high level concerns with taking this overall approach?

Regarding the implementation of stoplosses themselves (a problem that I hope I have now fixed), I am using SetStopLoss within OnBarUpdate at the time of entering a position, and using SetStopLoss to update the stoploss (say if the price goes up, the stoploss is moved up). I am ensuring that the SetStopLoss is always updated/reset before a new position is opened, however I read several posts that indicated that SetStopLoss may not be the best implementation of setting or updating Stop losses while a position is open. Is there a better/preferred method that I should be using for setting and updating the stoploss please?

Many thanks in advance

ChainsawDR

[NinjaTrader_Kate]

Hello ChainsawDR,

Thank you for your post.

I would concentrate on debugging why the strategy ends up in an incorrect short position rather than running a second strategy when the first shuts down - you can always just hit the close button to close the leftover position.

This sounds like potentially an overfill situation occurring with the stops causing an issue.

I would think about modifying your RealTimeErrorHandling to ignore all errors, add logic to handle any rejections, and then you can check if the strategy is in a Short position and simply submit an order to bring the strategy back to a flat position if so.

https://ninjatrader.com/support/help...orhandling.htm

However, setting RealtimeErrorHandling to IgnoreAllErrors can have serious adverse affects on a running strategy unless you have programmed your own order rejection handling in the OnOrderUpdate() method. You would need to monitor for any rejected orders and then resubmit or otherwise handle the rejection.

Please let us know if we may be of further assistance to you.

Edit: Just so you know, I've moved this to our Strategy Development forum since it's specifically regarding programming a NinjaScript Strategy.

[ChainsawDR]

Hi Kate,

Sorry for the long delay in responding. I'm still facing this issue and hoping to ask for some help in debugging it please as I am struggling. I've attached the logs for yesterday which had an issue (it was a single order made in the day). The strategy successfully went long in symbol JAN for qty 2600, entering at an average price of $3.89 with an initial stoploss of $3.03. When the stock price of JAN moved up beyond a certain point, it moved the stoploss up to $4.07, then $4.39, then $4.49 - all working correctly. From here though something went wrong - the stoploss was triggered when the price dropped and my 2,600 qty position was successfully exited, but my brokers API ended up receiving sell orders for a qty > 2600, which left active sell orders on the broker account that could have gone short if I hadn't manually cancelled, and the strategy stopped itself.

In the logs it shows the stop loss being updated but the Qty changing from 2,600 to 3,643, but I'm struggling to follow the meanings of the different states in the logs. Would it be possible please to cast an eye over these logs to see if you can understand what is happening here? To your original note, I'm very nervous about ignoring all errors and agree with your point of concentrating on debugging why the strategy ends up in an incorrect short position.

Note: My best guess had been that when the stoploss is triggered, some of the long position is exited but the price moves up again and the trailing stoploss gets updated, and updated with the original full 2600 qty - so I was thinking that I need to have a rule that says once the stoploss is triggered do not update it again. Unfortunately I'm struggling with the logs and don't see this explanation supported as I can't see the stoploss of $4.49 moving up, and I can't figure out why the qty changed +1,043 from 2,600 to 3,643 from the logs.

Thanks in advance for any assistance you can offer!

ChainsawDR

[NinjaTrader_Kate]

Hello ChainsawDR,

Thank you for your reply.

This looks like a problem with position updates not coming through before the strategy tries to move the stop. First of all, I do note you appear to be using Interactive Brokers - what version of TWS or IBGateway are you using to connect? You can find this on the splash screen when the one you're using starts up (often called the Build number). Next, are you on the latest version of the platform? The current version is 8.0.26.1. You can check this under Help > About.

Thanks in advance; I look forward to assisting you further.

[ChainsawDR]

Hi Kate,

Thank you so much for your help with this. No I didn't have the latest version installed, however when I first experienced the issue I did so I'm not sure if this is going to be it (but I will obviously update to the latest version!). Please could I ask - am I correct in thinking that overfill situations are always possible? E.g. there will always be situations where the broker executes an order but NinjaTrader doesn't yet know this and places a new order that causes an overfill situation?

If overfill situations are always possible, as I'm trying to get the strategy into a fully automated state (where I don't need to sit there and babysit it) I'd like to try and code for the scenario and exit everything in a reasonable timeframe, and wanted to check if the following approach seems sensible to you...

1. Modifying code for RealTimeErrorHandling to ignore all errors
2. If an error is received. Set a bool 'StrategyErrorExitFlag' to true and int '_lastErrorBar' to the CurrentBar.
3. Add a rule to the strategy that prevents any normal trading when StrategyErrorExitFlag == true
4. When StrategyErrorExitFlag == true, add logic that checks for current open/active orders with the broker every ~10 seconds (using _lastErrorBar) - if there are open orders, cancel them.
5. When StrategyErrorExitFlag == true. add logic that checks if the position is currently not flat, and submits a new market order seeking to go flat

Please kindly let me know if this sounds sensible. If so, if you don't mind I may loop back to check if the code I'm planning looks good to you (as I'm a little nervous about ignoring all errors)?

Thanks again for your help!

ChainsawDR

[NinjaTrader_Kate]

Hello ChainsawDR,

Thank you for your reply.

There's always a possibility of an overfill situation really, especially if there is a delayed response from the broker side and NinjaTrader doesn't receive an update on the position, it could certainly submit another order seeing the position is not closed.

We don't have a OverFill example on hand but there are a couple ways to implement overfill handling. In OnOrderUpdate() you could check if the iterating Order object has the OverFill bool set to true.

OnOrderUpdate() - https://ninjatrader.com/support/helpGuides/nt8/onorderupdate.htm

Another approach to handling OverFills would be to store temporary variables for your desired order sizes before submitting your order than to monitor the position size in OnPositionUpdate(). If you see a quantity filled that is greater than what you had stored for your desired size, you have detected an overfill.

OnPositionUpdate() - https://ninjatrader.com/support/helpGuides/nt8/onpositionupdate.htm

Handling the overfill would be subjective to what you wanted to do in the event of an overfill. (I.E. submitting another order to flatten the position or to stay in position and submit additional orders to protect the position that is still open.)

Please let us know if you have any additional questions.

[ChainsawDR]

Thank you Kate, super helpful. I like your idea of storing temporary variables for the desired orders sizes to monitor the position size in OnPositionUpdate(). Do you think this would work for both Orders & Trades? E.g. if a Long only strategy goes Short (trade made), and if a Long only strategy has an unfilled but active orders on the market (ordered but not yet traded). Apologies for so many questions, I'm not an experienced developer so trying to understand the theory first and understand where to research before diving in.

Thanks again

ChainsawDR

[NinjaTrader_Kate]

Hello ChainsawDR,

Thank you for your reply.

Something like that should work if you're in a long only strategy and there's an overfill situation where the strategy becomes short, you could check if the position is short and the position quantity is > 0 then do something, like exit the position. This won't work though for orders that haven't yet filled, however - they'd have to fill first to affect the position.

Please let us know if we may be of further assistance to you.

[ChainsawDR]

Thanks Kate. There being a 'bad' order out on the market, and waiting for it to be filled before selling is a scenario I'm keen to avoid. So in my strategy, for the attached logs, I went long for 2,600 qty (correct so far). I have only one section of code in my strategy that updates the stoploss, which is when the High[0] becomes greater or equal to my purchase price *1.1:

if (High[0] >= myOrderPrice*1.1)
{
SetStopLoss(CalculationMode.Price, Low[0]-0.01);
}

When I look through the logs I attached, I can't understand how the stoploss qty increased beyond 2,600 to Quantity=3,643. The error 'Cannot modify a filled order.' feels like a secondary problem, with the reason it set a stoploss qty greater than my position being the primary one. At line 99 when it states "Name='Stop loss' New state='Change submitted' Instrument='JAN' Action='Sell' Limit price=0 Stop price=4.49 Quantity=3,643" - do you have any ideas why the system would do this? I'm not specifying qty when updating the stoploss.

If relying on this, I was hoping to do something along the lines of:
A) Watching for any updates for "Name='Stop loss' New state='Change submitted'". If Qty > my original qty (2,600)
B) Cancel all open orders and wait 1-3 seconds or for confirmation.
C) Ping the broker to retrieve open position qty
D) Place a market order to exit the open position

My main questions please are:
A) Should I be updating the stoploss with a different approach?
B) Is there a method for the strategy to cancel all orders?
C) Is there a method for the strategy to 'ping the broker' to retrieve the current position size?

Thanks again for your help with this, it's really appreciated

ChainsawDR

[NinjaTrader_Kate]

Hello ChainsawDR,

Thank you for your reply.

Not a great way to monitor the stop/targets for incorrect quantities caused by partial fills with the approach you're using for setting stops and targets unfortunately - you're not saving them to an order variable so you can monitor them. I'd take a look at this example script that demonstrates using OnOrderUpdate and OnExecution update for tracking orders and positions - this approach may work better for updating stops as well since you can check the stop's properties before submitting a change:

https://ninjatrader.com/support/helpGuides/nt8/using_onorderupdate_and_onexec.htm

You'd need to be tracking orders like in the scenario above to cancel them just for one instance of a strategy. An example of cancelling active orders may be found in our help guide here:

https://ninjatrader.com/support/helpGuides/nt8/using_cancelorder_method_to_ca.htm

You could also consider using account methods to cancel all orders on a specific instrument or retrieve a position quantity but again, that would be for the overall account and not necessarily just the strategy if you have other things running on the same instrument and account (though we discourage running more than 1 strategy on an account/instrument at a time as this causes issues with synchronization)

https://ninjatrader.com/support/helpGuides/nt8/account_class.htm

As far as pinging the broker to retrieve the current position size for the strategy, you can't really ping them at will, NinjaTrader is dependent on updates from the broker to retrieve position information.

Please let us know if we may be of further assistance to you.

[ChainsawDR]

Thanks Kate, again this is super helpful. It's probably going to take me a week to study that example and review the documentation, but one thing that puzzled me straight away please in that example you shared for tracking orders and position:

// Reset our stop order and target orders' Order objects after our position is closed. (1st Entry)
if ((stopOrder != null && stopOrder == execution.Order) || (targetOrder != null && targetOrder == execution.Order))
{
if (execution.Order.OrderState == OrderState.Filled || execution.Order.OrderState == OrderState.PartFilled)
{
stopOrder = null;
targetOrder = null;
}
}

The comment implies that the stop order and target orders are reset to null after our position is closed, however it is accepting both Filled & PartFilled as arguments. If either Stop or Target orders were filled, then the position is closed so I understand resetting everything to null, but why would everything be reset after only a partial fill please? (wouldn't that imply that the position hasn't been fully closed?)

Apologies if I've misunderstood a concept here (as I mentioned, it's going to take me some time to study it properly).

Also sorry, can I please repost my question on qty: "At line 99 when it states "Name='Stop loss' New state='Change submitted' Instrument='JAN' Action='Sell' Limit price=0 Stop price=4.49 Quantity=3,643" - do you have any ideas why the system would do this? I'm not specifying qty when updating the stoploss." - In my strategy I'm letting NinjaTrader handle qty and just setting the stoploss, so I'm confused why NinjaTrader submitted a change with a different (higher) quantity that my position.

Thanks again

ChainsawDR

[NinjaTrader_Kate]

Hello ChainsawDR,

Thank you for your reply.

I'd actually remove the || execution.Order.OrderState == OrderState.PartFilled bit for your particular use case, since you still want to keep track while the order is filling.

As far as your quantity question, I suspect it's to do with StopTargetHandling.ByStrategyPosition. This may result in more stop and target orders being submitted than the overall strategy position in a scenario in which the strategy's entire entry orders are not filled in one fill, which combined with moving the stop is likely to have been the cause. If you use PerEntryExecution, I'd expect not to see this occur.


Please let us know if we may be of further assistance to you.