[Detrimental Request] Problem affecting seriously to Algorithmic trading

[ttodua]

As a continuation of my previous post (this also related) , I submit feature request (and wish many people voted too). In ninjatrader (after spent days in debugging the common issues with Algorithmic trading with strategies) we have found that there is nonperfect architecture of NinjaTrader's strategy's order handling functionalities (mainly in NinjaTrader.NinjaScript.StrategyBase class, all these problems can be fixed).

To repeat (what we have been researching in that referenced topic) the problem is that whenever strategy creates order and it is passed with the "Submitted" state order is into OnOrderUpdate (where we can detect this "Submitted" event), however after that the orders were lost without any further traces. We tested override OnOrderTrace() to monitor the orders and found there are MANY DIFFERENT SCENARIOS when the order action is not being passed to `OnOrderUpdate`. (The above referenced topic mentioned some example scenarios when this happens)

After enabling TraceOrders, it turned out that those orders' consequences were being revealed inside `OnOrderTrace` events (Thanks for that, we found that after "submitted" state, some logical criterias lead orders to cancel. HOWEVER, THEY WERE NOT BEING PASSED INTO `OnOrderUpdate` and we only detected that fact using TraceOrders).
However, there are 2 major problems for which we submit feature request and hope that you take seriously the vendor-programmers critical request, as many of our clients are affected for this reason in their trading.

(Problem 1)
TraceOrder logs are in PLAIN STRING Messages. Currently, that method just passes STRING MESSAGES:

protected override void OnOrderTrace(DateTime timestamp, string message)

which is surprising, why NT didnt do that, to provide `order` object (likewise OnOrderUpdate provides), instead of the string argument.
PLEASE IMPLEMENT A NEW OVERLOAD for that method to provide `order object` instead of the string message. AT THIS MOMENT, WE ARE DOING REGEX PARSING OF THE STRING MESSAGE TO UNDERSTAND THE REAL EVENTS AND ORDER DETAILS, to find out what happened to order and what is its status. This is so critically important, and just a ridicilous.

(Problem 2)
After you fix the problem1, that is not enough for algoritmic programmers. WE SHOULDNT BE BUILDING TRADING LOGIC for strategies which are dependent on `OnOrderTrace` events (we shouldnt be producing strategies which depends on customers to have TraceOrders enabled in their strategies, so as without enabling that, our strategies will just fail logically and lead to incorrect failure trading).
So, PLEASE PASS all events inside `OnOrderUpdate`. After debugging in Visual Studio, it seems there are many cases, somewhere probably within `SubmitOrderManaged` method, there happens OnTraceOrder events (giving logs that order was canceled due X reason), but then it returns from `SubmitOrderManaged` method right away. IT IS NOT CORRECT BEHAVIOR, because on every case of `OnOrderTrace`, there should be `OnOrderUpdate` events too, which will trigger our strategies `OnOrderUpdate` event, so we can control by our codes, whatever happen to order. PLEASE DONT ANYWHERE use `if (TracOrders) OnTraceOrder(..)` events without consequent `OnOrderUpdate` (or create alternative `OnOrderUpdateAll` overloaded method, which will be called always) so, we have every possible scenario controlled for order , and our orders wont be lost in abyss status and our strategies will not collapse. So, to repeat, offering to use `OnOrderTrace()` events to manage our strategies logic (to react when order is cancelled, ignored etc) is NOT A SOLUTION, as CLIENTS ARE NOT USING "TraceOrders=true" at all in their strategies and `OnOrderTrace` is not even a considerable method to build an entry/exit logic depending on it. `OnOrderTrace` is merely a quick debugger to output some messages, and cant be a LOGICAL part of strategy.

(Problem 3)
There are also many places, when NinjaTrader logs and prints (probably by method LogAndPrint) from `SubmitOrderManaged` however, OnOrderTrace doesnt happen. That is not correct also, as whatever logs are outputed into client's NT, those logs should be also available programatically in OnOrderTrace, so we have our hook to collect/analyze those logs, without SEPARATELLY grabbing LOGS from NT folders and then adding our TraceOrder messages, and thus, the combined bucket gives the complete scenario. INSTEAD, EVERYTHING whatever is passed in LogAndPrint (related to orders/submitodermanaged/etc) should be also available in OnOrderTrace events too.



Request:

I know that your response (whoever reads this) will be very formal "thank you for reporting" kind with undetermined unpromising reply, but please from NT SUPPORT DEVS, once again carefully consider that this is not a feature request for indicator, visuality or some candy features and attributes of platform, instead THIS IS DETRIMENTAL PROBLEM affecting MANY REAL TRADER clients of mine. You can check that we are NT Vendor programmers, and also we are providing services for NT programming to many SERIOUS clients (can be checked our profiles over different freelancing sites too) and we are meeting this problem already numerous times and our clients are affected for this, causing us to FAIL IN CODING the robust NT strategies and we are just suffering, while numerous clients are complaining about problematic orders several times in a day.

[bltdavid]

Can you attach an example strategy that could be used for testing?

[ttodua]

bltdavid unfortunately, the strategies I worked on are not mine personal codes, and couldnt share here their code (they are not simple, involving multi TF and some large logic, i.e. trading sessions etc... also, not easily reproduceable, as I said they happen ONCE IN A WHILE in specific situations, depending many different factors). The point here is that, even If i gave the example ( thought I will try in near future to create some example strategy), that might not be the big deal, as main problems arise from StrategeBase class and whatever happens in that class' internal logic, is known to NT devs only so, they will surely see that there are MANY cases, they can confirm what I say. Ideally, I think :
- there should not be cases when important action happens in OnOrderTrace[OOT] , but OnOrderUpdate [OOU] is not triggered (the reverse can be tolerated, i.e. when happening OOU, but doesn't happen OOT - that is ok, because OOU should be surely superior).
- there should not be cases when LogAndPrint happens, but doesn't happen OOT (actually, LogAndPrint provides some status change/failure/whatever important action about order, and OOT should be happening too with each case of LogAndPrint ).

So, that's why I wished NT addressed all those things. I have some doubts that they might never implement what I describe and they might leave OOU as it is now (so, ignoring the problem), but at least then, I would have suggested to create OnOrderUpdateAll (or smth) to be created, which will do the job and will not miss any occasion (so, will be including all events of OOU and OOT together), which will surely be superior and replace any need to use OnOrderUpdate.

[NinjaTrader_Jim]

Hello ttodua,

Thanks for your post.

As you are aware feature requests have no promise of fulfillment and interest is tracked and weighed against feasibility before considering implementation. Requesting a feature to workaround a problem that you are experiencing is likely not the way you would want to move forward. There also is not any guarantee that these proposals would resolve your issues.

The underlying problem I see here is that your strategy is losing track of orders. We would not expect this and would want to understand how the platform is involved in simple terms. If it is a bug with the platform, we would want to get that case addressed instead of submitting a feature request and not looking into the matter further. If it is a bug in your strategy, reducing the strategy to the issue at hand will help to resolve it.

OnOrderTrace handles the string that is printed in the Output Window when TraceOrders is enabled. TraceOrders should really only be enabled when testing a strategy, and you have already confirmed that the order submission methods have been reached in your logic. (It is most useful to tell us why order submissions have been ignored when using the Managed Approach.) It should not be used as a logical part of the strategy. Printing your own debug info in OnOrderUpdate is possible if you want to focus on the orders life cycle.

We would like to focus on the problem and this will require a test script that can reproduce the issues. We do not want a full strategy, we only want enough code to demonstrate the issue in simple terms.

I suggest first testing to reproduce the issue with your strategy so the issue can be encountered consistently. I then suggest to reduce the strategy code by sequentially commenting out code and get to the specific functionality that is causing issues. We may also suggest modifying the strategy logic so it takes certain orders on certain bars as opposed to generating a signal from indicators or your proprietary logic.

From there we would have a small example that we can use to reproduce the behavior on our end, comment, or file a bug report with Development if that is the case.

We look forward to assisting.