Total Net Profit very different- optimiser display vs Trades display

[tgn55]

Thanks Jesse...

I never expected NT Support to create a case that displays the error... nor to debug my code, we are on the same page on all of this...

In this case, however, where on the one screen, Strategy Analyzer shows two completely different result sets in different panels...I find it very hard to believe that it is due to a bug in my strategy, and not something more fundamentally broken in the reporting code.

It is the "different result sets" thing that convinces me the underlying reporting code is broken. I spoke about this at length in my post #7... without re-running the strategy, one panel shows results using one set of strategy parameters... the other panel shows results from running the strategy with a DIFFERENT set of parameters! Sorry ... that surely cannot be my code at fault...but seems it has to be a bug in the underlying mechanism for regurgitating stats. That is the crux of the matter.

Of course... a repeatable process to trigger the issue is highly desirable... which in due course I will try to provide, but it won't be using SampleMACrossOver! Yes, my strategy code that revealed this issue is fairly complex (albeit based on MA crossovers)... and (hopefully) has no grievous sins committed in it! But barring memory leaks or something like that, I'm not sure it matters how complex my strategy is. If "the system" is showing two different result sets on the same window in different panels... I think the developers need to be looking elsewhere for the root cause.

Enough of this ... we will only conclude this if/when I can provide the test case... but I am still ;-( consumed by trying to resolve a complex Infragistics multi-level grid display issue, so it might be a little while yet, sorry!

Cheers, and thanks for the excellent support you provide... I really do value it (even if I seem a bit grumpy at times!)

T.

[NinjaTrader_Jesse]

Hello tgn55,

It is the "different result sets" thing that convinces me the underlying reporting code is broken.

We don't need to look that far into the issue. Knowing if something internal is working or not is not relevant. If you see a problem we need to work toward finding the steps to reproduce that and what code was used.

Support has no ability to look into the internal code, we cant see that code. We can only collect information and report it to development who can look into that code. Going into detail as you did in post 7 about suspicions is not going to help move forward on the issue. What will help is creating a reduced sample and steps to see what you see so I can make a bug report or at least see what code was used to know why that's happening.

Of course... a repeatable process to trigger the issue is highly desirable... which in due course I will try to provide, but it won't be using SampleMACrossOver!

That's fine if you don't use that specific script, but you need to make it simple like that to report an issue. We can't debug your code so if the script you provide is complex and doing other things which are not required to see the problem its going to get rejected. Making a simple example will come by debugging the code to find what controls seeing the problem and not seeing the problem.

You previously mentioned that the SampleMACrossOver does not have the problem so that would make a easy way to build up to see the problem. That eliminates your complex code and requires you to extract the problem thus reducing the code at the same time

Yes, my strategy code that revealed this issue is fairly complex (albeit based on MA crossovers)... and (hopefully) has no grievous sins committed in it! But barring memory leaks or something like that, I'm not sure it matters how complex my strategy is. If "the system" is showing two different result sets on the same window in different panels... I think the developers need to be looking elsewhere for the root cause.

It does matter because if you use code which does not make sense or inherently has errors it may cause problems when being used in various tools. Debugging the code is very important, in most cases reducing the code to find the problem will allow you to just fix the problem with that code. If you find that you are using something which should work we can take that code sample and make a bug report to see why its not working like it should be.

[tgn55]

Thanks Jesse, but with respect, you are still totally missing the point.

Please, go and take a closer look at the picture I included in the first post. It shows in the top section a total profit of $54K for the selected row. The section below... Trades, shows the final Cumulative Net Profit of only $35K.

Here is another even more extreme example... taken from a completely different strategy... no code in common with the original... and the reported results appear at first glance to be utter nonsense.
NOTE: I get this inconsistency when I pull previously run results from the SA LOG tab... I believe that is where the problem is (but read to the end...)

The huge discrepancy in reported profit is staggering... and CANNOT be a result of my strategy's code... it is just how Ninja reports a prior optimization run.



So... 8 small trades giving $62 total profit... turns into over $6000 profit, as reported by NT's SA system. If you think that can be due to a bug in my code... well, I'm amazed. Please... stop trying to blame my code... it patently is NOT. My code does not create the above report .. internal NInja code does... and it is clearly broken.

<T does some more investigation...before returning to write more stuff>

Now... the good news... I think I have solved the mystery of the root cause (see, I'm not such a dummy after all ;-)
It seems that when data is pulled from the Log tab... NT does NOT obey the specified Default Quantity... nor Include Commission (and who knows... maybe other setup parameters as well??) when reporting the Trades... but obviously does when reporting the overall profit. Very inconsistent, guys...

So, in the pic above, each trade results in a few bucks P/L... but I had Default quantity set to 100... the difference in reported profit in the top section DOES include the qty 100 factor (and commission)... the reported list of trades has Qty=1... and NO commission.

I am not sure if this also explains the weird numbers in my original post... I no longer have that row in my Log so I can't easily check... but back then, I was mostly trading futures, typically Default Qty would have been 1... not 100, which I only recently used testing a new strategy on stocks. But... the internal reporting logic is just wrong... and when the workaround of re-runnng the optimization in a new tab is applied... the setup qty parameters etc ARE applied. Problem solved (at least in this instance)

This is a bug...

PS... Yes, it does the same with SampleMACrossOver ;-)

Cheers,
T.

[NinjaTrader_Jim]

Hello T,

To be clear, we are not saying the issue is with your code. We would need to reproduce the issue with a simple strategy or the SampleMACrossover strategy to rule out extraneous possibilities where a strategy may be affecting the matter, or at least get down to what is in the code that is related to the issue.

As you mention that you can hit this with multiple strategies and the SampleMACrossover strategy, this does look like a NinjaTrader issue.

In order for us to report the matter we need to reproduce the issue and we have not been able to do so. Below is a recording of my test where I have attempted to reproduce the issue.

Demo - https://drive.google.com/file/d/1HLj...w?usp=drivesdk

If you can provide me written steps to follow or a short demonstration video like mine so I can follow and hit the issue, we will have what we need to make a report. If the steps require a custom script, please reduce/simplify it so it only has the code necessary to demonstrate the issue.

I look forward to hearing from you.

[tgn55]

Hey Jim... Just saw your vid... but as I'm in Australia, it is 1:40 AM, and I'm going to bed!

I will create a video for you tomorrow, the process I followed is very close to what you performed... you should have it by Monday. Try selecting "Trades" in the DIsplay thing, rather than "Analyze", after you pull a historical optimization from the Log tab list... that might be all you need to do to show it... anyway... my video will demonstrate it

I don't see the image I included.. I hope you got that ... it really does make it abundantly clear . I'm going to attach it on this post, rather than embedding it, to be sure...
You can see the Qty 1 in the Trades display...and no commission as discussed above.

Cheers,
T.

[tgn55]

OK... I have made a video.. but also found a condition I was previously not aware of. The problem bites AFTER restarting the platform... not sure if it happens ONLY after restart.

In this video I firstly show the most recent optimization failing after restart... but if I click on any other row in the Log history that had Default qty > 1... I get the same result, ie inconsistent display of performance between the various panels on the report screen.

Demo: https://drive.google.com/file/d/1AK7...ew?usp=sharing

I know the trades data is NOT saved in the strategyanalzerlogs xml file... it needs to be recreated by rerunning the strategy... but some key parameters (like qty) are ignored, and so the resulting trades list is (possibly) in disagreement with the summary results.

So... if/when strategies are reviewed from the Log tab... ignoring some parameters like this is bound to cause confusion, and lack of confidence that the results reported are indeed reliable. If the strategy really is rerun to generate the trades list... it would be a good idea if it used ALL setup parameters...not just some of them.

PS... Best to view the video full-screen to see the content more clearly...

Hope it helps,
T.

[NinjaTrader_Jim]

Thanks T,

I reproduced right away with the help of the video. I will keep this thread updated with any new information as this gets reviewed internally.

[tgn55]

Thanks Jim.

I may never have nailed this had I not changed to stock trades, with qty=100 ... which kinda made the issue extremely obvious. Prior to this, eg in my first post - the difference was significant... and unexplained, but not so obviously a quantity multiple. What I had observed was seemingly random occasions when things were out of synch. So now we know why;-)

Moving on....

[NinjaTrader_Jim]

Hello tgn55,

The issue has been recognized as a bug by Quality Assurance and has been assigned to the Development team to work.

The ticket ID is NTEIGHT-14891, and the number for the ID will be listed in the Release Notes page of the Help Guide when a new version is release with a fix for the issue.

Release Notes - https://ninjatrader.com/support/help...ease_notes.htm

Please note that release schedules are determined by Product Management and development tasks and we cannot offer an ETA on new releases.

Thanks for your report!

[tgn55]

Thanks Jim... I look forward to seeing this one nailed.