Strategy Analyzer Log tab - View Strategy errors out

[NinjaTrader_ChelseaB]

Hello T,

If the code of a strategy is changed, especially the inputs, may I confirm you have deleted the old logs?

May I confirm you are able to reproduce the behavior after running a new optimization and creating a new log?

[tgn55]

The xml file is not corrupted... the SA Results pane shows the contents of the xml file correctly. (Yes, I checked...) So typically we have 10 rows of summary data shown in the top pane.

But, selecting one of those rows re-runs the strategy (new code) which may generate results that are different to what is saved in the xml file under strategyanalyzerlogs. No corruption needed!

I should be in bed... now 5:00 AM !! Tomorrow... the full story with pics, movies etc...

[NinjaTrader_ChelseaB]

Hello T,

Changing the strategy code, especially public inputs can corrupt any xml file. This includes workspaces, templates, and StrategyAnalyzerLog files.

If these code is changed, it is recommended to delete any old logs, workspaces, or templates and resave these.

Regarding the Commission, are you able to reproduce this after running a new optimization with the Include commissions checked in optimization parameters, and then re-opening the newly saved log file?

[tgn55]

Hi Chelsea...thanks for the video you recorded... but I think I may have tracked down the cause of this behaviour... but it's now after 4:00 AM where I am, so might leave a full write-up until tomorrow !!

Briefly... I see (sometimes) odd results in the Strategy Analyser window... which you will NEVER see just testing Sample MA Crossover... because... I CHANGE the code of my strategy as I am developing it ... and of course, you are using static, unchanging strategy code.

What I've observed tonight is that when I select a different row in the Results (top) section of an optimisation, it re-runs the strategy (knew that from our earlier discussion about where the list of trades comes from... which is NOT pulled from the XML file ;-). However... here's is the crux of the matter (I think...)

If the code has been modified since the optimisation in the log tab history... you might get different results. In my case, new code has a parameter that simply was not there in the earlier version. When the strategy runs, that parameter gets set to zero... I checked in Visual Studio, set breakpoints etc so I can examine what is happening. And in my case, this results in a different number of trades being placed, and a significantly different Net Profit etc. So, as I've said from months ago... you end up with a huge (and puzzling) mismatch between the row in the Results pane, and the results reported in the Summary pane!!

OK... making progress... but I have one thing I am seeing that has me stumped....

In the Strategy Analyser settings pane, I have Include Commission - true. But, when I trigger the strategy to be rerun (by picking a different row)... the trades show zero commission... and when I check in Visual Studio... IncludeCommission is false... !!!! And how that gets modified, I really have no idea.

Anyway... more tomorrow. I'll send some screenshots at a minimum... maybe some video.

Cheers,
T.

[NinjaTrader_ChelseaB]

Hello tgn55,

I've given this a test but I am not able to reproduce.

Below is a link to the test I have just performed.

https://drive.google.com/file/d/1VIVYbrhtjI_-BR-iNE74SEnva1IqaINQ/view?usp=drivesdk

If you take the exact steps shown in the video are you getting different behavior?

[tgn55]

NT 8.0.20... ID NTEIGHT-13990 is claimed to be fixed ... but is NOT fixed. Clicking on W/F rows, and then selecting rows from the Summary table at top of screen, shows completely out-of-synch results. By which I mean things like number of trades, and Total Net Profit reported in the top frame are nothing like the numbers shown in the Summary View. I have no idea where it is pulling the info from... but it is CLEARLY bogus. still.

I did not Open in New SA... can't see why I should need to. The existing SA window is displaying nonsense. Maybe opening a new window is a work-around... but there is still a bug that causes the existing SA window to show god-knows what trades when you select different rows from the top pane...

And BTW... this invalid data is NOT restricted to walk-forward optimisations. If I view a normal optimisation from the log, I also get bogus results. ie, the number of trades, Total Net Profit etc reported in the Log history are completely different to the data shown in the Summary display. That.. in spite of other things (like Settings) appear to be correct.... so it is pulling SOME data correctly... but the really important bits... NO!!! Completely and badly wrong.!!!

[tgn55]

Great - thanks for following up.

I finally did get to test myself the "Open in NS Editor" thing ... yes, it does exactly what I was looking for a way back, so that's a win... Thanks.

[NinjaTrader_ChelseaB]

Hello T,

I have received tracking ID# SFT-4286 for this request for the View Code button to also work with backtests and optimizations.

Please note it is up to the NinjaTrader Development to decide if or when any request will be implemented.

We appreciate your feedback on this.

[tgn55]

Hey Chelsea, you know I never noticed that option before! This whole thread started when I discovered the "View Strategy" things... (which incidentally is what I was referring to being greyed out now) which I understood would open the cached version of the .cs doc at the time the back-test was performed... terrific, just what I wanted ;-)

The Help guide included above does not refer to HOW you get to view the code... just says you can do it. I found what I believed was the method by poking around... probably just browsing the options in the right-click preferences view.

You know, I'm going to have to follow this up tomorrow... it's after 3AM where I am, and last night i was up the whole night... yep... traded straight through. This may be the solution I am looking for... but I need some shut-eye !!

Thanks,
T.

[NinjaTrader_ChelseaB]

Hello tgn55,

Run a backtest.

Show the Logs.

Right-click the log result for the backtest -> select Open in NinjaScript Editor.

Is this the functionality you are referring to that is mentioned in the help guide?

[tgn55]

ID 13842 is only partly fixed... or I might say fudged ;-) The buttons in the Log history are now greyed out so no object ref error occurs. But... the functionality is still missing! Ie, I am saying these buttons should NOT be greyed out... just make them display the strategy! Should be a no-brainer, as the code is sitting on the analyserlogs folder already...

Confirmed the bomb on right-click on SA Log History heading issue ID 13961 is fixed in 8.0.19.

[tgn55]

Hi, following up after starting to test 8.0.19... ticket 13842.

View Strategy buttons... so they are now greyed out in the Log tab. This is not really a satisfactory fix...
Please see the image I attached in reply #13, which clearly states that code snapshots ARE available in the log... and NO MENTION of AI Generated-only.

I know the code snapshots are in the strategyanalyzerlogs folder... so it should be real easy to make the "View Strategy button work! The code is there... just need to pull it into the editor when the button is clicked !

Image from Ninja documentation repeated here...



I can't easily check this, but I'm pretty sure the above wording existed looooong before AI-Generate was even a twinkle in a developers eye ;-)
That said... I recognise we may be talking two different things here...not clear if the View Strategy buttons were a recent addition in 8.0.18 along with AI generate...

So... can you please re-open the ticket, or raise a new one to get this sorted... and make the behaviour match the documentation?
Or I can raise it as a new feature request...

I'll reply to the other issue separately after more testing.

Thanks.

[NinjaTrader_ChelseaB]

Hello tgn55,

NinjaTrader 8.0.19.0 has been released.

Please update to the latest version and confirm the behavior can still be reproduced.

[tgn55]

Chelsea... I have an update on the issue of incorrect data being displayed.

1. It is NOT restricted to walk-forwards... plain old optimisations also do it.
2. I have positively confirmed that the incorrect performance data (and it is VERY incorrect) that is displayed is absolutely NOT anywhere in the XML files under the strategyanalyzerlogs folder
3. We already confirmed that each trade is NOT saved in XML... but generated afresh by re-running a backtest. OK...
4. If I open an optimisation in a new tab... the right (?) stuff is displayed


Conclusion has to be... the invalid data initially displayed is coming from memory.. and is very wrong. Careful inspection/comparison of the two reveals that they show a similar pattern of wins/losses (although the PnLs are quite different, sometimes by a huge amount) UNTIL the session close (which in my timezone is 6:30 AM) Then it just goes crazy for the next several hours. See pics. I think the dramatic change after session close is significant. Lastly, there is a different number of trades !! So it is not simply getting the PnL wrong/different on each trade... it goes deeper than that.

As said, opening this in a new TAB fixes it. But, the initial view presented to the user without jumping through any hoops is plain wrong. (or maybe the other view is wrong...) And no walk-forwards involved.

This was most graphically illustrated when I did some looking at analysis by exit time, as I just modified my strategy to use "Exit on session close".

This is the initial view immediately after optimisation: (Please see the notes I inserted in the pics...)


...and here is the SAME backtest displayed in a new tab...



Wow. What a difference!!
And, BTW... the Summary performance total net profit stat... $14K in one... $22K in the other.

Sigh... hope this helps developers to find what is going screwball.

Cheers,
T.

[tgn55]

Ahhh... OK. Wondered why I couldn't sniff them out! I did read that somewhere... but as I was getting results back really very quickly as I moved around the log list, for example if you have a pane showing trades, then click on a different row, it seemed the data must be cached somewhere. Maybe my computer is faster than I thought !!

Thanks...