Bug in NinjaTrader: SA shows incorrect MFE in certain cases

[bltdavid]

Hi Betrand,

Under Account Performance -> Trades -> MFE column.

I sold two contracts in the ES at 2099.50 and exited contract 1 at my first PT at 2097.25 and got stopped out of my runner at 2096.25. On both contracts, however, the MFE is 6.75 in the MFE column. So apparently, the MFE for the first contract is based on the movements of my second contract after the first contract is closed, which obviously makes no sense, leading to flawed statistics and conclusions.

Thank you for your help with this.

I betcha more information will be needed ...

Were you trading from a chart or from a strategy?

Were you using the sample strategy I provided?
[If so, Bertrand will surely respond that your results are expected behavior, same as he did with my results]

Were you trading manually? Live account? Sim account? Backtesting?

It is important to know how you entered the initial entry order for the 2 contracts.

I have not tried to investigate the correctness of MFE values in other order entry scenarios.

So, I'm very curious, what scenario are you reporting?

Can you reply with exact steps to reproduce?

Thanks for looking closely at your MFE values ...

[swetrader]

I betcha more information will be needed ...

Were you trading from a chart or from a strategy?

Were you using the sample strategy I provided?
[If so, Bertrand will surely respond that your results are expected behavior, same as he did with my results]

Were you trading manually? Live account? Sim account? Backtesting?

It is important to know how you entered the initial entry order for the 2 contracts.

I have not tried to investigate the correctness of MFE values in other order entry scenarios.

So, I'm very curious, what scenario are you reporting?

Can you reply with exact steps to reproduce?

Thanks for looking closely at your MFE values ...

Sim trading ES from chart. 2 contracts limit sell order (recorded as one order with 2 contracts in the order log).

I'm totally new with NT and I simply cannot understand why this seems to be the default way of calulating MFE, since there's just no logic to it.

[bltdavid]

Sim trading ES from chart. 2 contracts limit sell order (recorded as one order with 2 contracts in the order log).

I'm totally new with NT and I simply cannot understand why this seems to be the default way of calulating MFE, since there's just no logic to it.

Hmm, yeah, I think I see what you mean. This is indeed a new & different scenario that produces the same incorrect results I found using m/m methods in my attached strategy.

[This kinda reinforces my contention I had with Bertrand & Cal in an offline email. I argued that this incorrect MFE issue was dire enough that it should have nothing to do with how it happened (in my case, via "abusive" use of just 2 m/m methods in a simple strategy). I mean, the numbers are flat wrong, so why does m/m matter so much that m/m makes these wrong numbers "expected behavior"? By association, that phrase also means acceptable behavior. Nonsense, the numbers are wrong, they are unacceptable, plain & simple. MAE and MFE have industry accepted definitions, after all.]

Ok, back to what I found ...

I, too, see incorrect MFE values trading on the chart, same as swetrader, using a quick and dirty test with the "Simulated Data Feed", using 2 contracts (target+runner).

Here are my steps:

1. I disconnected from my live feed and reconnected to "Simulated Data Feed". [Why? Markets are after hours right now, so price is barely moving, but with SDF I can control "trend" and force price to move in a certain direction. Cool feature.]

2. I bring up a new ES chart using 5-Minute bar series. I enable chart trader, but select <None> for ATM templates. [Why 5-Minute? Because NinjaTrader will adjust the price axis to show more price points, so I have more room for my test orders above and below current price. I skip using any ATM because I'll right click my mouse and do all my order management right on the chart.]

3. On the chart, I right click below current price and select "Buy Limit 1". After the order appears in the chart window, I click on the "1" and change it "2". This sets up a 2 contract entry for target & runner.

4. Then 8 ticks below my 2 contract entry order, I right click and select "Sell Stop 1", again after the order appears, I click on the "1" and change it to "2". This sets up the stop loss for target & runner, which would be $200 total loss if it actually gets hit, $100 loss for each contract.

5. Now I adjust the "trend" slider in the tiny "trend" window, so that price drops to my 2 contract entry order. Bam, I'm in.

6. Then 4 ticks above my entry price I right click and select "Sell Limit 1". This is the target order, so I leave it at "1" for 1 contract. 4 ticks is a $50 profit target.

7. Now I adjust the trend slider to move up and hit my 4 tick target. Bam, it's hit.

8. After hitting this 4 tick target, I let price continue to rise while I click on the "2" for my stop, and adjust it down to "1". This stop is now the runner's stop order only, and so the quantity is changed to 1.

9. Now I adjust the trend slider to move down, and take out the stop. Bam, it's hit. I'm now flat.

10. End of test, disconnect from SDF, click on "Account Performance" tab, then "Trades" tab, and Generate a report showing the 2 trades.

Expected results:
1st trade (target) should show a 4 tick profit of $50 with MFE of $50.
2nd trade (runner) should show an 8 tick loss of $100 with an MFE > $50 (because I let price wander higher after hitting target before moving the trend slider down to take out the runner's stop)

Actual Results:
Both trades show same value for MFE, and this value is > $50.
Both trades show same value for MAE, and this value is the $100 stop loss.

Now see attached screenshot.

--Screenshot Summary--

1. The MFE for the 1st trade is clearly wrong. It should be $50, same as the profit column. Why? Because a limit order caused this trade to exit, so this trade could not possibly have seen an MFE any higher than the $50 profit from the limit order's exit.

2. The MAE for the 1st trade is also clearly wrong. Why? Because this trade did not dip as low as $100 before hitting it's target price. If it had, it would have hit the stop loss of $100 (8 ticks), which was the stop loss setup for both target & runner.

3. Look how trade #2 shows a loss of $100 in the profit column. This makes perfect sense because this trade (runner) hit its 8 tick stop loss. Look how it's MAE is also $100, which is completely correct; the worst MAE this trade ever saw is the same as the loss generated by the stop order.

4. Look how trade #2 shows an MFE of $125, which corresponds to my test case where I let price roam higher after hitting the $50 target on trade #1.

5. Clearly the 1st trade did not have an MFE of $125. The best MFE it ever saw was the $50 of profit generated by the target's limit order.

6. Clearly the 1st trade did not have an MAE of $100. The worst MAE it ever saw could not be the same $100 stop loss of the 2nd trade, because this would imply the target+runner exited at the same price (which would only happen if the stop loss is hit *for both* contracts at the same price and at the same time, which would have only happened if price never rose to hit the target, but instead price dropped to take out the entire position via the stop).

There we have it, the numbers are utterly and irrefutably wrong.

Q.E.D.

So, in summation, I implore the justices of the court to consider my pleas,

1. The 1st trade's MAE/MFE numbers, in this scenario, are incorrect.
2. The behavior of NinjaTrader, in this scenario, is incorrect.

These incorrect values should *not* be expected behavior of an m/m strategy.
These incorrect values should *not* be expected behavior of chart trading.

We see the same pattern of incorrect values in both scenarios. And, both scenarios are extremely simple, universal, and easily reproducible.

Ergo, this is absolutely, unequivocally, a bug in NinjaTrader.

To claim anything else, to make any attempt to justify these numbers as "correct", or "acceptable", or "expected behavior", is simply illogical, completely indefensible, and deeply irresponsible.

Phew, I think I've made my case.

[bltdavid]

Btw, it should be obvious that, for the trades where the MAE/MFE values are wrong, the ETD is also wrong.

Why? Because when a trade exits at its profit target (so that its MFE is *supposed* to be this profit target amount) there was no money left on the table in the trade. Thus the end trade drawdown (ETD) should be zero.

See also:

http://www.ninjatrader.com/support/forum/showthread.php?t=1021

[bltdavid]

I had said this in a earlier post:

The trades in the TradeCollecton have no support for the concept of a "position" consisting of "1 or more trades" which would justify the MFE for all trades on all lines of that position to be made the same.

Strictly speaking, part of this is not quite true, and potentially misleading to anyone following this thread closely.

A TradeCollection *does* have support for the concept of a "position" consisting of "1 or more trades", see here,

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?gettrades.htm

but this API stops at the fetching of Trade objects, it does not calculate MFE values.

In fact, if you look closely at the Trade Object, see here,

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?trade.htm

You'll see that a Trade object does not expose properties for the MAE/MFE/ETD of each trade. I presume this means NinjaTrader has to calculate these values on-the-fly when displaying the Trades tab.

[My intuition says that, if these 3 values were already calculated before hand, then the fine engineers in Denver would have exposed these values right there in the Trade object, so that us mere mortals can access them, too.]

So, I'm guessing *that* is why we see the same pattern of incorrect values for a simple m/m strategy as we do for a simple trade on the chart. The code producing those values is *centralized* and only calculates those values *on-the-fly* whenever needed, such as when displaying the Trades tab.

My point is: (and I'm still guessing here) if this code had stored the values after a one-time calculation, naturally the best place to have done that would be the Trade object itself, but the MAE/MFE/ETD are not *exposed* there, so I presume they're not *stored* there either. [I'm not quite the whiz with Visual Studio or Reflection, so I can't say I know how to peek at all the properties & methods that are exposed but simply not documented. Still gotta learn more about that ...]

Anyways, I'm just saying, we see incorrect results regardless of chart trading or strategy trading -- that is a clue that the code creating these incorrect results has nothing to do with m/m, or chart trading, per se.

My forensic programming skills tell me it is highly likely the MAE/MFE/ETD values are centrally computed by the Trades tab, on the fly, when needed, probably from data (and only from data) available in the Trade collection. Thus, this code doesn't know anything about an m/m strategy or chart trading -- it simply goes by the Trade objects in the collection.

Ah, and there's the rub: The Trade object does not have pre-calculated and pre-stored MAE/MFE/ETD values as a courtesy to the Trades tab, and thus, darn it, that code has to do its own math, and whichever early programmer did that math got it wrong, presumably, for some very simple and common scenarios.

Am I right?

Lol, maybe, maybe not. Witness that the Summary tab's AvgMae, AvgMfe, and AvgEtd *are* exposed and available for the entire collection, so why aren't the individual trade Mae/Mfe/Etd values that comprise these averages (and shown on the Trades tab) exposed as well?

[bltdavid]

For information about AvgMae/AvgMfe/AvgEtd in TradePerformanceValues, see here,

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?tradesperformancevalues.htm

[bltdavid]

Hi Bertrand & Cal,

Any updates on this issue?

Is NinjaTrader able to confirm this issue qualifies as a bug?

Best Regards

[NinjaTrader_CalH]

David,

Bertrand is still working on this however he has been unavailable for a time and will get back to this as soon as possible.

Thank you for your patience.

[bltdavid]

It's been almost 2 weeks since my last inquiry.

Any updates on this issue?

Is NinjaTrader able to confirm this issue qualifies as a bug?

[NinjaTrader_Brandon]

Hello bltdavid,

Thanks for your reply.

The representative working on this unfortunately is still out and it is unsure when they will return. We will review your case and get back to you on this.

[outsource]

I guess the Sharpe ratio metrics is calculated wrong as well.Though i`m not looking at it, but still,for those who are - be advised.

[NinjaTrader_Brandon]

Hello bltdavid,

I have reviewed the ticket, tested, and replicated what you are reporting. From my tests and reviewing documentation it seems that these value within NinjaTrader do in fact group these results together as 1 strategy to complete the trade. I will submit to development to separate these, taking each exit as it's own trade. Thanks for reporting your findings on this.

[bltdavid]

Excellent news, thanks for the update!

Is there a tracking number for this bug?

[NinjaTrader_Brandon]

Hello bltdavid,

Thanks for your reply.

I do not have a tracking number for this however I would expect that this will be looked into for NinjaTrader 8.

[ganamide]

Does this mean NT 7 has reached it's end of life? It seems to me that it could still be quite some time before NT 8 is ready for release.

I'm astonished that after so many years nobody noticed these problems in NT before. I am also seeing problems in MAE, MFE, and ETD calculations on NQ futures in Account Performance, Trades tab. I enter trades using Chart Trader with an ATM strategy, so they all have the same Entry Name. I see impossible values even for single contract trades. Maybe NT is not using tick data to calculate these statistics?

Furthermore, I am seeing that Sim mode is filling me at impossible prices. Either that, or the times are off by a few seconds. Today I even had a Sim limit target order not get filled even though price clearly spiked through it! I'm starting to lose confidence in NT as a reliable trading platform, at least when it comes to Sim and Trade Statistics.