Trade Performance Report is wrong...

[tgn55]

Thanks Jim... but I respectfully don't agree.

I believe the problem of incorrectly paired executions occurs if/when the first execution found in the selected report date range is an exit. The entry may have occurred minutes, hours or even days prior... I don't see how millisecond resolution of execution timing has ANY bearing on that.

In my case, I got "sensible" trade reports on (at least) one instrument... by changing the report start date to a point where I was flat on GC or whatever it was. My point above is that fiddling the start date is NOT adequate to guarantee success, in a multi-instrument scenario.

If I am missing something - forgive me... but can you please explain why my suggested solution, ie, when assembling TRADES from executions, in a given period, one should ignore exit executions until you find an entry execution? It seems very logical to me...

Regards,
T.

[NinjaTrader_Jim]

Hello tgn55,

There is more than one limitation at hand. The start of the performance report is important and can cause misalignment with trade pairing, but in your first post we can see that the timestamps for the execution match when we start to see the pairing deviate. I.E. the Close Position and SE orders show executions with the same timestamp. SE is listed first and Close Position is second which makes the report skewed. (There are other instances of this in the screenshot, but this is one example.)

If millisecond granularity is noted with the executions, NinjaTrader could properly identify which came first and correctly pair the executions.

Your suggestion for having the trade report ignore Exit executions until an entry is made would be plausible for the issue with performance reports getting skewed from a starting Exit execution, but the issue with trades pairing incorrectly from executions with the same timestamp would still be prevalent. Looking in our enhancement request database there is another enhancement request for having the trade report look back to find associated entries. Your request is similar but I have noted you are requesting to have the report skip Exits and then to start with a known entry from within the report. Ticket ID SFT-3514.

We are aware that there are limitations with how trade pairing works, and we are tracking impact and feedback for users experiencing these symptoms. Symptoms could be common in some limitations, I.E Trade Pairing is off, but the cause for each case may be different and would require a different change.

Let me know if you have any additional questions.

[edstar]

Where is the list of SFT up for voting, I'd like to see the list so I can vote on other issues which may be of interest, there needs to be more transparency. In the mean time you could at least provide an example of how to add a manual execution to flatten a position (with screen shots) so we could at least do that in the interim until the fix is implemented, I note this has been outstanding for a number of years.

[tgn55]

Jim, I confess I did not recently go back and look at the first post... the second post re my GC trades with the 4 pics sort of subsumed the first. And in the post #5, what is evident is that the day boundary of the performance report starting on 22nd falls mid-trade.

I was short from 11:54 PM on the 21st... closed that and went long at 12:02 AM. So, the performance report beginning at midnight sees the exit from a prior position, and builds a "trade" starting with that exit. And that is just plain dumb, leads to nonsensical "trades", complete nonsense MAE/MFE numbers... a whole slew of garbage follows from simply starting a trade with an exit...and THAT is the crux of my previous rant ;-)

In the situation where a exit/entry happens within milliseconds...as may happen when switching from short to long(?)... I think is a separate issue... and which may indeed require fine resolution timing to sequence the execution log properly into trades. I get that... if NT processes the new entry before the close at "about" the same time... well, things will still be out of synch, so I concede there is value in the millisecond thing (but see below)

But it is not always the cause of garbage performance reports! And my plea is to get the terribly obvious "clangers" addressed by skipping initial exits (I still suspect scanning backwards will have more complications...eg if scaling out of a position etc) Any crossover strategy may suffer this unfortunate possible timing issue... but, other strategies, or manually placed trades will not... and it seems not unreasonable to at least get it right for those!

Now... some deep-dive stuff...
Even when a strategy does a "swap short for long" thing, resulting in the two trades at 12:02:40 AM on April 22nd...(as reported in Trade Performance report)... there is ALREADY enough information in the database to ensure correct sequencing...

a) the execution IDs... but, even if they are not guaranteed to be strictly in sequence... then
b) the time of trade! I can see (by delving into the sdf) recorded trade times of

637230763609510000, for execution ID 303587803661 (the short close) followed by
637230763609700000 for execution ID 303587804429.(the long open) Clearly, a later time!

.....................^

This is the "raw" data, The bit AFTER the caret above shows the fractions of a second... ie, This is internal representation for the two times April 22, 12:02:40 AM (plus a tiny bit ;-)

So... seems to me the millisecond resolution information is ALREADY available in the database ;-)
Why then can the trades not be correctly constructed?

Sorry to be a pain in the rear, but as I see it, this leave little wiggle room. The developers just need to a) NEVER start a trade with an exit, and b) use the existing sub-millisecond time of execution.

I look forward to your further input...

Finally: Note to edstar: if you scroll back, there is already a reference on how to add remove executions. But... I would seriously advise against it!

[tgn55]

Apologies for the length of the previous post.. I was sort of figuring out stuff as I went along, investigating sdf dumps etc.

In hindsight, it could have been summarised much more succinctly:

• Execution times are already recorded to the millisecond in the sdf
• NT in many cases that do NOT involve such hair-splitting, STILL starts "trades" in the Performance Reports at stupid places...with an Exit.

NinjaTrader_Jim... looking forward to hearing more from your side. If I can help with anything, let me know...

Regards,
T.

[NinjaTrader_Jim]

edstar, Feature/Enhancement requests are tracked internally and are not publicly available. (We include internal notes and discussion within those tickets that we do not wish to share publicly.) New Features/Enhancements are noted in the Help Guide when available.

tgn55, I had a chat with Product Management regarding trade pairing in general and chatted about the matters presented here.

From a base line, we should understand that NinjaTrader is a front end application. If we are trading intra day and NinjaTrader sees all executions that come in, we can expect the trade pairing to work as intended.

Some limitations that may be seen:

1. If NinjaTrader does not see an execution, this will leave a a gap that will result in mismatched trades. This would be expected since NinjaTrader cannot know every trade that is made on the account. For example, you could close NinjaTrader, close a position externally, and then NinjaTrader would not see that execution.

2. If there is a position that is left open overnight, this will skew the report (from Trade Performance window or from the Executions tab of Control Center) as we would see an exit execution start the report. This would be an expected limitation, however, we have tracked your interest in having NinjaTrader leave out exit executions if those start the report. As you can imagine, some users may not want to have the exit execution filtered, and would just want to see the executions that NinjaTrader has seen. This could be optional, but nonetheless, the idea is tracked, and you could open a Trade Performance report starting from a point where you are flat to have consistent trade pairing.

3. Executions of the same time stamp will also cause pairing issues, however, what you are stating with the millisecond granularity would be correct. After clarifying with Product Management, the trade pairing will use the timestamps with the granularity of what is stored in the database.

4. A separate issue that we may see is if the Exit is listed after the Entry in a Data Grid. This would instead be a bug related to the Infragistics grid in which a later execution would be listed before an earlier execution when sorting by time. This would not affect trade pairing but would have to do with how the grid gets sorted.

Regarding #1 and #2, these cases would be expected to be encountered with NinjaTrader's design. Some fixes can be thought of to address these limitations, however, we should consider that since NinjaTrader is front end and does not have full access to the executions/trades that the broker logs, there will always be cases where what NinjaTrader reports will not be accurate to what the broker has logged.

Regarding #3 and #4 if we see discrepancies here, I.E. we can run a Trade Performance report from a spot where we are flat and the trades are displayed out of order, or are mismatched, we would like to review further. Please then email use your database file and a screenshot showing how you have set up the performance report. The Control Center's Help > Email Support option can be used to send in the database file.

We look forward to assisting.

[edstar]

Jim, I can understand there are different circumstances as you have mentioned, some users are holding positions and some of us are day traders intraday. I think it is also necessary to understand the root cause these mis pairing are happening, from my observations there are 2

1) NT crashes during an open position
2) A data connection loss occur for a prolonged amount of time during an open position.

If we could get an alert at least which

1)checks the log for a connection loss and also
2) when NT8 starts up that quickly scans the DB and alerts us to the situation of where the the last execution is of an Entry per instrument / account

Then at least a user can knows to check the DB and can manually delete the last entry.

I think these are an easy step to lessen the pain for the short term.

A more automated solution may take more time.

[tgn55]

Thanks Jim... several comments required.

If we are trading intra day and NinjaTrader sees all executions that come in, we can expect the trade pairing to work as intended.

That's not strictly correct... due to timezone differences as I've already mentioned.
What may look like an intra-day trade to you... may look like a trade that crosses a date boundary to me. The example I gave demonstrates this clearly, I think, but I'll say it again...

If I place a trade say at the open, 9:30 EST... and close it a couple hours later, then in my timezone that will look like an entry say on late Monday night... and exit on Tuesday morning. Boom... Trade Performance report starting on Tuesday will see the exit as the first execution, and everything from there on will be screwed up. UNLESS... my suggested policy was implemented - which would fix it.

Now... your limitations section:

1. Missed executions... well, yeah... but this has nothing to do with the issues I and many other traders encounter. So I acknowledge that technically this could happen, and if it did, Ninja could not do anything about it... but really, it's not relevant. It is NOT the cause of the rubbish performance reports I can generate, so let's put it aside.

2.Sorry, I have said several times already... saying

you could open a Trade Performance report starting from a point where you are flat to have consistent trade pairing

just doesn't cut it, and I have explained why... if I have several instruments I am trading, going to the inconvenience of identifying when I am flat in one instrument (say, GC), may very likely coincide with a time when I am NOT FLAT in another instrument, and hence this proposed solution is a waste of time. It WILL NOT WORK CONSISTENTLY. My suggestion will. Always. For all users, no matter what timezone they are in.. No matter how many instruments they have open trades in at the start of the performance report. Seriously... what's not to like about THAT ???
And finally,

some users may not want to have the exit execution filtered, and would just want to see the executions that NinjaTrader has seen

Again this is NOT the point. If users want to see all executions - they can !! Just select "Executions" in the report.. not TRADES.

3. Identical time executions... I don't know if this is even possible at millisecond granularity, but in any case... MOST if not all issues of pairing would be resolved by correct sorting of executions to the millisecond, plus my main strategy of skipping initial exit executions.

4. Don't care about the position shown in the execution grid... I agree executions made within milliseconds may show in the grid in the wrong order (but seriously? If the developers can't sort these also on actual execution time, rather than displayed time (which is only to the nearest second... you should get better developers ;-) But the key thing is... this will NOT impact reported trades.. if all else is managed sensibly, so it too is a bit of a non-issue as far as fixing the broken TRADES report.

NinjaTrader_Jim ... as far as I can see, my solution would PERMANENTLY put this ongoing issue to bed, never again to trouble the users, the forum or the developers. I just can not understand why Ninja continue to resist implementing it. I can't see any of the points you have raised constitute a valid reason not to do so.

Regards,
T.

[NinjaTrader_Jim]

Hello tgn55 and edstar,

We are aware of the pain points with the current limitations and we have heard your suggestions. Our Product Development group has a way to solve the matter in mind but it requires server backed storage of all trade data.

This concept is on the list of things to navigate in the future.

[tgn55]

Thanks Jim... but I am surprised. I'm pretty sure I could knock up some code that pulled the executions out of the (local) sdf file and create a kosher trade report. In fact, when I get a bit of spare time, I might just do that! But... it's a tad annoying that I have to.

I just can't see what issues might prevent this, and certainly cannot see why I would need "server backed storage of all trade data", given all my executions are in the sdf file. Anyway... maybe I have a surprise waiting for me!

If I succeed... maybe Ninja can offer me a job as a freelance developer ;-)

Regards,
T.

[tgn55]

Hi all
I have just written up a lengthy post on this topic under this thread: https://ninjatrader.com/support/forum/forum/ninjatrader-8/platform-technical-support-aa/96261-grouping-trades-in-trade-performance/page2

Bottom line is, I have implemented a fix., not quite ready for release, but should not be too far away.
And the solution is as I have been saying... skip initial exits. ALWAYS.
I probably need a little more testing yet, (plus I want to see if I can also address the issue of grouping trades in the report), and REALLY need to see if I can make it run more efficiently.

That said... it works as expected, and creates sensible TRADES reports.

Please take a look at the other thread, I don't want to repeat it all here... but I do plan to continue any further discussion in THIS thread...

Regards,
T.

[marialewis16]

I probably need a little more testing yet, (plus I want to see if I can also address the issue of grouping trades in the report), and REALLY need to see if I can make it run more efficiently.

That said... it works as expected, and creates sensible TRADES reports.

T.

Any luck with this? Seems like you are our only hope

[tgn55]

Hi, I have not been able to do much work on this for the last month.. but other issues have now mostly been put to bed, so I might have a chance to finish it off.
I also built a desktop app that implements the same algorithm... ie outside of Ninja. I don't think that's a great solution... we users need the internal Performance thing FIXED... my effort is really just to demonstrate a) that it can be done, notwithstanding NT's assertions to the contrary, and b) how to do it right!

I will post an update when I have a final product... if you want, subscribe to this thread so you will be notified.

Thanks for your interest!
T.

[tgn55]

Well, it's been a long while since I provided an update... my apologies, but the need to make some money pushed this down my priority list!

I spent AGES creating an Addon that does all this (and more... see the link to the post re grouping trades), and my what a world of pain that opened up! Largely due to some UI features that do NOT adapt to changes in selected skin... but I digress.

I have created an Addon that:

• Never confuses entries with exits
• Does not require the user to mess about with report start dates etc in futile attempts to prevent nonsense results. It just works...
• Allows for trades to be grouped, so, scaling in or out can be treated as 1 logical trade rather than multiple trades if you wish to report it that way
• Has selectable first-in/first-out or last-in/first-out logic
• Display explicitly which entries and exits were matched to produce each trade (no guessing)
• Correctly manages unusual execution sequences, such as where you might switch from being Long 3 contracts to Short 2 contracts (by selling 5 contracts)
• Provides more novel ways of displaying the results with advanced filtering built in
• Reports any anomalies discovered during processing in a log

I confess I have not worked on this for the last several months... but that's pretty much it. The bad news is...I still don't quite have it ready to publish (that will require more messing with Addon packaging that most of you just don't want to know about)

Having spent literally many hundreds of hours on this (yeah, I got nerd-sniped bad...) , I hope to sell it on the Ninja ecosystem... I guess I am wondering how much interest there might be... which could motivate me to do the final prep so it's ready to ship.

Let me know if you'd like more information...
Regards,
T.

[marialewis16]

Well, it's been a long while since I provided an update... my apologies, but the need to make some money pushed this down my priority list!

I spent AGES creating an Addon that does all this (and more... see the link to the post re grouping trades), and my what a world of pain that opened up! Largely due to some UI features that do NOT adapt to changes in selected skin... but I digress.

I have created an Addon that:

• Never confuses entries with exits
• Does not require the user to mess about with report start dates etc in futile attempts to prevent nonsense results. It just works...
• Allows for trades to be grouped, so, scaling in or out can be treated as 1 logical trade rather than multiple trades if you wish to report it that way
• Has selectable first-in/first-out or last-in/first-out logic
• Display explicitly which entries and exits were matched to produce each trade (no guessing)
• Correctly manages unusual execution sequences, such as where you might switch from being Long 3 contracts to Short 2 contracts (by selling 5 contracts)
• Provides more novel ways of displaying the results with advanced filtering built in
• Reports any anomalies discovered during processing in a log

I confess I have not worked on this for the last several months... but that's pretty much it. The bad news is...I still don't quite have it ready to publish (that will require more messing with Addon packaging that most of you just don't want to know about)

Having spent literally many hundreds of hours on this (yeah, I got nerd-sniped bad...) , I hope to sell it on the Ninja ecosystem... I guess I am wondering how much interest there might be... which could motivate me to do the final prep so it's ready to ship.

Let me know if you'd like more information...
Regards,
T.

Great news! I appreciate the time and effort and am looking forward to this!!