Total Net Profit very different- optimiser display vs Trades display

[tgn55]

Hi,
can't explain this one... I am reviewing some optimisations I did recently, and noticed discrepancies... sometimes fairly huge, between the TNP reported by the optimiser and the Cumulative Profit shown in the Trades list. Example shown is fairly extreme...

I'm inclined to believe the Trades list... which also agrees with the graph of Cumulative Net Profit. It would be reassuring if they both matched the data shown in the top 10 list!

And yes I am showing the last of the 22 trades made, so cumulative profit should be equal to Total net profit...
I found a similar post ... but it was way back in 2009... I hope things have changed since then !

See attached pic

Emphasize: this occurs when reviewing runs saved in the Log tab. Seems to work fine on a fresh optimisation... so maybe it's in the logic for pulling results out of the database?

Regards,

[NinjaTrader_Jesse]

Hello tgn55,

Thank you for the post.

You said this happens with saved results specifically, can you see this happen with the SampleMACrossOver strategy?

I tried a few quick tests to see if I could recreate a difference in saved results, but I was unable to see a difference. Does this happen if you open the saved result after another test, for example run a test, run another test and revert back to the saved result?

Could you provide more specific steps of the process you followed when you saw this?

I look forward to being of further assistance.

[tgn55]

Hi Jesse,
tried with Sample MA... all looked fine. I then went back to my ZB log entry ... and it also looked fine! But... I then noticed that I had run another optimization on GC in between. The wonky results came from my earlier run...

So, I have two different optimization runs on ZB in the log, with a GC run in between. But same strategy, same parameters. One is kosher... the other isn't.
I actually exported the trades showing the $35K trades profit the other day, so I was able to now export the trades with the correct (?) trades, at least what matches the $55K Total Net Profit.
Same number of trades, but some exited at different times/prices. On examination, I suspect I can partly explain... I refer to IsRising on an SMA, which in all three cases where the trades differ, the SMA is pretty flat. So, maybe a comparison of floating point numbers thing is going on...

However... that surely does not really explain why the reported trades in both cases show nearly $55K profit in one pane... and $35K in the panes (although that does match the trades). Surely it should be internally consistent? I'm seeing a a strange mix of the two trade collections!

Can I email you the Excel docs I have? It would make it clearer, I have highlighted the differences. Only 22 trades, and a few differences, so not too onerous to check.. if you wish, although I'm not sure it really helps decipher the mysterious reporting.

Regards,

[NinjaTrader_Jesse]

Hello tgn55,

Thank you for the additional details.

I don't believe the excel documents would be helpful at this point, the image you provided clearly shows the difference.

What I can pull from this description is that you ran the ZB -> GC -> ZB, do you always see that happen when doing tests between different instruments in that order?

You also noted the SampleMACrossOver seemed fine, if that is the case it is possible that the problem is isolated to your custom strategies logic in some way.

From this point, I would suggest that we try to find a situation where your strategy can more or less reliably fail in reporting the values. Once that situation is found we can confirm if that happens with the Sample again to make sure the problem is or is not isolated to your script. This may be one reason I am unable to see the problem at the moment.

If this happens to be isolated to your script, you can run some additional tests by reducing the scripts logic. Making the script more simple or temporarily removing parts of its logic can be used to tell if that is part of the problem or not. Another approach is to comment out your logic, and copy the OnBarUpdate/variables from the SampleMaCrossOver to see if the script now works. These type of tests could be used to further extract the problem if it is something in the logic.



I look forward to being of further assistance.

[tgn55]

I'll see what I can do Jesse. I don't really believe it's my strategy logic... which is a kind of crossover, pretty basic, and I have reasonable experience coding. I'm always keen to nail any nasty gremlins hiding in NT, I'll probably return to this next week. Tonight (I'm in Australia, its already after 11 PM ;-) I'm going to be focusing on making the strategy more robust... bottom line, I will come back to follow up... but probably will not be tonight!

And I'll ensure I capture exactly what I do ... assuming I can reproduce the weirdness!

Cheers,

[tgn55]

Hey Jesse... me again. Just reviewing things... it's been a couple months since I posted this, I got kinda distracted with other SA issues...

I have just posted another issue with dodgy results in the SA area,

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1069244-strategy-analyser-profit-per-month-inconsistency

Not clear if they might be related, I can't really see how... but thought it might be worth cross-referencing just in case.
I have not yet got any further with this particular anomaly... I am already spending about half my time submitting bug reports! Will try to return to this issue and see if I can create some further info.

Meanwhile... try the above link. It's pretty whacko stuff...

Cheers,
T.

[tgn55]

So... I have continued to be plagued by these inconsistent reports periodically since I started this thread 18 months ago. Sigh... So this is what I've figured out/observed.

1. Trades are NOT saved in the sdf file.
2. The Strategy Analyzer does NOT rerun a strategy necessarily every time you click on a row in the Log pane ... and I suspect this is key to what causes the erroneous reports.
3. Right-click>Open in New SA Tab DOES rerun the strategy... which is probably why this "workaround" mysteriously "fixes" reports that are just completely out of synch... ... but I digress
4. I have test cases in which it is abundantly clear that the results such as shown in the pics in my post #1 actually come from running the strategy, seemingly with the same parameters... but, in fact... using a DIFFERENT parameter set. (My strategy has LOTS of parameters, but trust me... the results are extremely obvious that the results shown purport to come from one set, when they really come from a different set... I will in due course reveal all... but, the purpose of this post is to try to zero in on a test case using your beloved Sample MA Crossover ;-)

OK... sorry for the lengthy preamble.

My SUSPICION is that when results like pics shown in post #1 arise, the results MUST be pulled from memory, since they are NOT stored on disk AFAIK, and, as I have print statements in my strategy that confirm if/when it is run... I KNOW my strategy is NOT rerun... so... seems clear the numbers must be getting pulled from cache somewhere. Memory, or disk that is NOT the sdf...

Next step ... and again, I am guessing... and looking for confirmation/denial... I SUSPECT that identifying the results to pull from said cache are keyed based on things like strategy name, start/end dates, and... ??? at least SOME parameters??? I am surmising that, if a match is found in the cache on effectively a SUBSET of the parameters, then that block of results is displayed... albeit possibly NOT the correct results for applying ALL parameters in the Settings pane. At least, that is what I am seeing with my strategy.

So.. wrapping up... if you can confirm my beliefs (or correct me), I will attempt to create a test, if possible, using the old MA X standard... I say "if possible", as if my guess as to underlying mechanism is somewhere near the money, the vanilla MA X thing may NEVER actually trigger this behaviour. Anyway... I have some ideas on how to address that, if it is the case.

Apologies for making you read a novel... but if I am right... this might result in fixing a long-standing and VERY significant bug in NT...

Regards,
T.

[NinjaTrader_Jesse]

Hello tgn55,

My SUSPICION is that when results like pics shown in post #1 arise, the results MUST be pulled from memory, since they are NOT stored on disk AFAIK, and, as I have print statements in my strategy that confirm if/when it is run... I KNOW my strategy is NOT rerun... so... seems clear the numbers must be getting pulled from cache somewhere. Memory, or disk that is NOT the sdf...

I couldn't really say as we are referring to internal logic there. From your and my perspective where the results come from doesn't matter, we really just need to understand the steps that lead to that result so we can report it.

Next step ... and again, I am guessing... and looking for confirmation/denial... I SUSPECT that identifying the results to pull from said cache are keyed based on things like strategy name, start/end dates, and... ??? at least SOME parameters??? I am surmising that, if a match is found in the cache on effectively a SUBSET of the parameters, then that block of results is displayed... albeit possibly NOT the correct results for applying ALL parameters in the Settings pane. At least, that is what I am seeing with my strategy.

Again I couldn't really comment here as that's going to relate to internal logic which we cant view. Knowing where the results come from really does not help to find a solution or to report the problem. If you isolated some steps that reproduce what you had described that is what you should focus on or providing the most simple example of those steps. We don't need to go into actually debugging the why its happening, just how to make it happen so development can review it.

Please let me know if I may be of additional assistance.

[tgn55]

Hmmm... well, I can show the results from my strategy, but you know the usual response... "show it in Sample MA Crossover" I was hoping you could provide something that would help me do just that...

OK... I will post a follow-up (maybe tomorrow... it's now 3:30 AM where I am ;-) with pics (maybe a video) showing what I see on my complex strategy (at least in terms of parameters)... and will TRY to replicate in Sample MA X. But I fear I might be chasing the wind, if things do indeed work the way I suspect.

Cheers,

[NinjaTrader_Jesse]

Hello tgn55,

Hmmm... well, I can show the results from my strategy, but you know the usual response... "show it in Sample MA Crossover" I was hoping you could provide something that would help me do just that...

OK... I will post a follow-up (maybe tomorrow... it's now 3:30 AM where I am ;-) with pics (maybe a video) showing what I see on my complex strategy (at least in terms of parameters)... and will TRY to replicate in Sample MA X. But I fear I might be chasing the wind, if things do indeed work the way I suspect.

Right, we don't have the capacity to debug custom code so using the sample or a very simple strategy is the recommendation if you are trying to explain an issue. If your custom code is having a problem that will help to set a baseline so you can better understand the problem. If the sample can show the problem then we have a bigger issue that involves the platform and we can most likely just report it. If it requires some specific set of syntax and steps then making a simple example is the quickest way forward. We would need to know what that syntax that was and how it was being used before we could provide any insight or fixes.





Please let me know if I may be of additional assistance.

[mlprice12]

Did this issue ever get resolved?

I am having a similar issue right now and can't seem to figure out a solution. Total Net Profit is not equal to cumulative net profit for all iterations. Sometimes they match up and sometimes they don't. This is an optimization for my instrument alone.

In the attached photo iteration 1 has the same net and cumulative while iteration 2 is off.

[tgn55]

No... I have spent the last 6 (or 9?) months working on an Addon that addresses other significant NT reporting issues... and it has turned into a bit of a monster. Hoping to get it out "sometime soon", but to be honest, the last 2% of the UI code has taken longer than the rest of the project!

But, back to this issue... I have not done further testing since my last post, as I REALLY need to finish off my reporting Addon first. I have just resorted to the workaround of opening a new SA tab or window... it fixes the incorrect data, but is not really satisfactory, since one never can be 100% sure if what you are seeing on screen is legit... and constantly opening a new tab "to be sure" really... well.. sucks.

Apologies... but it might be a while before I go back and look again at this. Maybe we just need to do as Jesse said... produce a repeatable example of it producing nonsense numbers... and HOPE the developers do something about it...

Cheers,
T.

[mlprice12]

I appreciate the work T.

I ran into the same problem. I'll be on the lookout for a repeatable example and send it down the pipe.

[tgn55]

Jesse... or other NT staff: I think the bigger issue here is the company position that unless an issue can be demonstrated with Sample MA Crossover strategy then THERE IS NO PROBLEM. Restated, this amounts to the following equivalent assertion: Sample MA Crossover is sufficient to test ALL and EVERY possible code path in NT's strategy analyzer and associated reporting system.

This is clearly and patently obviously an absurd assertion.

What I initially reported is clear evidence that something is massively wrong... not just a little bit wrong. Yeah, I get it... NT users might have any number of weird bugs in their code, but when a reasonably advanced coder gets these kind of errors, with NO exceptions thrown, and when the symptoms clearly indicate the reporting system is pulling data from the wrong instance of a backtest..operation... then relying on the defense ":Well, it works fine for Sample MA Crossover" just does NOT cut the mustard. And unless/until NT 's staff accept that more complex strategies can and very likely will reveal bugs that will NEVER be exercised by the most basic test... nothing will change... and significant bugs in the product will likely never be addressed. Which is a real shame.

We users are trying to help NT improve the product by providing valuable feedback... not just get spoon-fed solutions to our needs or have misuse of the platform corrected, which I recognize is probably about 98% of what the support folk do, and is appreciated... but there is more.

Anyway... stepping down from soap-box...

Regards,
T.

[NinjaTrader_Jesse]

Hello tgn55,

Without being able to demonstrate the problem to development I would not be able to form a bug report, a demonstration of the issue is a basic requirement to even start a report. Without that information we just don't have the resources to try and develop a custom script that should match your description of the problem, there are a great number of variables involved in custom code which may be different from what you used to produce the issue. We very likely would be unable to produce the problem only with a description, that is a bad way to approach the problem when you already have the real code and situation which you can use to make a demonstration.

We also cannot debug custom code for you so using the existing samples to make a simple way of displaying the problem is always the suggestion, that avoids the complex script situation and makes sure you have debugged your custom code yourself. The SampleMACrossOver makes a good starting point for you to add what you need to see the problem and only that code, not a lot of extra unneeded code.

If your code is really complex we just don't have the resources to go through that nor is that a service we can provide to debug said code. Reducing the code to the minimum to highlight the problem would be the basic expectation for any code related bug reporting, if we don't know what specific code was used to see that result we really don't have anything to go on other than your description. In some cases a description is enough, for example using the user interface and seeing an error. With code that is often not going to work, we would instead want the specific code you used so we can address the problem in the correct way and not make assumptions or guesses.

Please let me know if I may be of further assistance.