Strategy Optimizer Results tab not the Sames as Real Results

[Robinson94]

Hi, i hope you are doing fantastic !

I'm currently facing a little issue with the use of the Optimizer in the Strategy Analyser.

It is around the Results window (so after an optimization, the window that will display either the results of all or of the top X number of optimization for the specific data series); the issue is that, the Results (so amount of trades, PnL and everything) are totally different from the Result window, to when i launch the script on a chart based on the same data series, or simply when i click on the specific optimization on the Result Window. (with the exact same parameters)

I have seen one thread about this subject, yet the issue came from the fact that the person was using random number generation, which i am not ( https://ninjatrader.com/support/foru...ad.php?t=98666 )

Using or not using the "IsInstantiedOnEachOptimization" makes no difference in the randomness of the Results window. I do reset in the script (in DataLoaded) every single variables, order object, series, session iterator, etc; i also reset the values of the indicators in each indicator in their state dataloaded. (i put them to 0, false, recreate the new series, or DateTime.MinValue, yet i do not set the values of int and double to MinValue like shown here : https://ninjatrader.com/support/help...niteration.htm )

The script is quite extensive (+20000 lines), so i would mostly like if you have any ideas of where those random results in the Results Performance of the Optimizer could come from, since when opening a specific result in the optimizer (from what i understand it re-run the test), i have always the same values and no randomness, same as when running it on a standard optimization or chart; that i use no random creation in the script, that i reset every variables for the strategy and the indicators in their State DataLoaded, and that i do not make any difference to the way the code work when in optimizer or not (unless to the use of a special case to keep using a session iterator, yet this logic is used in the standard strategy analyser, which work just as expected).

I would gladly appreciate some help, since this issue make it not possible to use the Optimizer with the script, it can still run, yet, since the results are not correct, the "top" parameters set for the specific data series, are also not correct (and change each time i relaunch the optimization).

Thank you so much for reading this quite long post, have a great day !

[NinjaTrader_PatrickH]

Hello Robinson94,

Thank you for your post.

When you run the same test in the Chart as the result of the Optimizer are you ensuring the same number of bars (date range) is used?
Do you see the same behavior with the SampleMaCrossover or you already know it is due to your strategy' logic?

When you state the following can you detail what variables are "reset"? Are you referring to the optimized parameters?

I do reset in the script (in DataLoaded) every single variables, order object, series, session iterator, etc; i also reset the values of the indicators in each indicator in their state dataloaded.

I look forward to your response.

[Robinson94]

Hi again !

I just tried on the SampleMACrossOver, it is working well on there, so its a positive that the issue come from the strategy logic or something specific to it.

And when running the script on the analyser i will use a date range, and on a chart, i will use a end date with the amount of days, which at the end, create the exact same data series.
Also note, that since the issue is only happening on the Results Window of an Optimization, when i open the chart of either a Standard Strategy Analyser result, or a specific Optimization Result, then the trades, logic, data series, and everything is the same as on the other places.
So it is only on the Optimization Results Window that the issue occur.

When i said reset, i intend all the variables that are created inside the strategy and indicators as private or public variable, and that are not NinjaScript Property (like shown on the IsInstantiatedOnEachOptimizationIteration page). And i do not actually reset the values of the Indicators, yet i reset the Values of the Variables of the Indicators inside their own State DataLoaded, the same way it is done for the strategy.

If it is not that precise, i reset in State DataLoaded (set to 0, false, DateTime.MinValue and others) all the variables that are created before we open the OnStateChange void. This is done both in the strategy and indicators, and there is no addon involved.

So, i do not reset in any way the optimized parameters, or any NinjaScript property.

PS Edit : Also i noticed in the logs, that, at the very start of the data series, before the strategy had taken any trades, yet when it was starting possible for here to take trades, that the message "Calculated Stop price was smaller/equal, not stop placed" has popped a few times in 35 minutes (of the data series, not 35 real minutes), even if there was no position, and it will not do that when not in the optimizer.

Thank you for the incredibly quick respond !

[NinjaTrader_PatrickH]

Hello Robinson94,

Thank you for your response.

Can you attach a screenshot of the optimization results in question as well as a chart with the same settings showing differences in the results?

To send a screenshot with Windows 7 or newer I would recommend using Window's Snipping Tool.

Click here for instructions: http://windows.microsoft.com/en-us/w...#1TC=windows-8

Alternatively to send a screenshot press Alt + PRINT SCREEN to take a screenshot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save as a jpeg file and send the file as an attachment.

Click here for detailed instruction: https://www.java.com/en/download/faq/screenshot.xml

I look forward to your response.

[Robinson94]

Hi !

Could i send you a short video by mail to be able to show you the issue more prescisly ?

Thank you !

[NinjaTrader_PatrickH]

Hello Robinson94,

Thank you for your response.

Please reach out to me at platformsupport[at]ninjatrader[dot]com with 'ATTN: Patrick H' in the subject line and a reference to this thread in the body of the email.

[Robinson94]

Issue fixed and solution

Hi !

I wanted to share that i fixed the issue, and how it has been fixed.

First, thanks for the great help of Patrick, that helped me a ton in pointing me to the right direction.

Basically, the issue was coming from a bad reset of variables on each iteration, even if i was actually resetting all the values, the same way showed on the IsInstantiedOnEachOptimization page of the Help Guide.


The catch (which took me tens and tens of hours to figure out, at least 30-50), was all simply that i was using static variables for my script !
And unfortunately, from what i read, its not as easy to reset static variable, than classical private or public variables.

Yet, i needed to be able to share data between my scripts, and i could with ease, in an indicator for example, have a public variable, that the strategy could access, and all would be good on there.

Yet, if i wanted for an indicator to get data from the strategy, i would had to use static variables (since unlike the strategy, the indicators do not load the strategy), and the static variables cause the issues in the reset on each iteration.

So the fix was, that if an indicator need to get data from the strategy, instead that the indicator will directly go get the data from the strategy trough a static variables (NinjaTrader.NinjaScript.Strategies.MyCoolStrategy .MyCoolVariable), i will create a public variable in the indicator, then, inside the strategy, i will set the value - that i want the indicator to get from the strategy -, in the public variable (not static), that is in the indicator, and then i can use in the indicator the value from the strategy; yet it is actually the strategy than send the value, and not the indicator that retrieve the value, the indicator is receiving the value.

Also, if you are using a parameter in order to add an indicator or not, be sure to check that you actually added the indicator in the strategy (by using the same condition as the one for adding the indicator), before trying to send data to the indicator, otherwise you will have a error.


Maybe this post can help somebody with a similar issue, and also, even if now, i know (like it is commonly know in the C# community), that static variables do not work and reset the same as not static variables, it is not mentioned in the Help Guide on the IsInstantiedOnEachOptimization page, maybe it could be a good idea to add a little note about that ?

Have a fantastic day !

[NinjaTrader_PatrickH]

Hello Robinson94,

Thank you for your response.

Can you provide an example of the static variables that would not reset?

I would like to look into this matter further.

[Robinson94]

Hi !

The use of any static variable would have not been reset properly (even if the variable was not affected by any other script), no matter if it was a public static or private static variable.
I'm talking about static variables created inside the Strategy script.

The issue would happen, even if the optimized parameter, was only a dummy parameter that did not influence the script, yet, if it was a dummy parameter that was static, then it would not cause issues, since the static variable would not be used.
So it was not coming from the creation of a static variable, but from the use of a static variable in the logic (since it would not properly be reset).

I did not fully understand the way the static variables work, yet, i did read that the way it reset is totally different, since it do not only exist in the script, but kind of in the entire program (i'm not at all an expert on this as you can see).

The weird thing, was that the static variable would do get reset at the start of each iteration, yet would still not reset properly.

Anyway, the only lead that i can give you if you dig in this, is that the use of a static variable in the strategy script (public/private static MyCoolVariable), would create issue, even if they did show like they got reset on each iteration.

I have no more infos about this issue, maybe it can help, yet, what i read was that the best solution, is simply to avoid at the maximum using static variables, and that a lot of people really didn't like them; and just following this advice, and finding a way to make it work without static variables, is in my opinion the best way to go.

PS : I had also try, to instead of using the static variables in the script logic, i would used not-static variable, and then set the value of the not-static variables to others static variables, that would be used for sharing the data; it had not fixed the issue.

Have a fantastic day !