Strategy Analyzer Stops Iterating Parameter Selections During Optimization

[ChazJ]

As I run the Strategy Analyzer in optimization mode with various parameter settings I have noticed that it will get into a state where the selected parameters are not being exercised.

As an example, I have a strategy which uses moving averages and I have a parameter to select the type of moving average so I can see which one works best. When I first run it, it works fine. However, later, after playing around with some other parameters, I can see that the Analyzer is no longer going through the list of moving average types. It is always selecting the default. The results tab shows that the settings are different for each run, but the results are identical. This is what led me to question the operation. I then verified this using Visual Studio and setting break points as it is running. It always uses the default as it is running.

If I close Strategy Analyzer and re-open it, proper operation is resumed until some later point in my testing. I recently upgraded to build 20. I am not sure if this was happening in build 19. I have to assume it was. So, obviously, I have to now invalidate all my previous results of testing, which has cost me a lot of wasted time.

Please let me know what other info I can supply you with to troubleshoot this issue.

Thanks,

[NinjaTrader_Jesse]

Hello ChazJ,

Thank you for the post.

I was unable to locate any current issues which match this description, we would likely need to see an example of the syntax used along with the settings used in the optimization when you see this to see if this can be replicated.

For testing I would suggest using the SampleMACrossOver to see if you are able to replicate it with that logic. If the sample cannot reproduce the problem this may relate to something specific being done in the custom strategies logic.

Can you additionally check the log tab when you see this happen to check if any errors are being reported?

I look forward to being of further assistance.

[ChazJ]

Hi Jesse,

I think I have found a way to reproduce this issue with the SampleMACCrossOver strategy, but I had to modify it to add the parameter selection for Average type. I have attached the modified sample.

The item that causes the issue to occur is the line:

// Calculate = Calculate.OnBarClose;

in the OnStateChange method under State == State.DataLoaded

Run this sample, then remove the comments from that statement and run it again. The Debug.Print statement will show that the same average type is being used for each instance of the optimization run. The strange thing is that the Calculate value is already set to OnBarClose, so this statement should be having no effect. Nevertheless there must be some code behind the property setting that is causing this issue.

I hope you are able to reproduce it using this code.

Regards,

[NinjaTrader_Jesse]

Hello ChazJ,

Is there a reason this is being set from DataLoaded and not SetDefaults? This is not expected to ever be toggled later in logic so I would not suspect this would be a problem just wrong placement of where the property being set.

If you instead use State.SetDefaults do your tests succeed?


I look forward to being of further assistance.

[ChazJ]

Hi Jesse,

I was prompted by the Help Documentation located here: https://ninjatrader.com/support/help.../calculate.htm

where it states:

"Typically this property would be set in State.SetDefaults, however scripts that require Calculate to be set by the developer must set this property in State.Historical in its OnStateChange() in order to ensure that if this script is a child (hosted) that the parent.Calculate property which is adopted by the child is overridden again."

I had moved it from State.Historical to State.DataLoaded figuring it would be better to set it earlier. In any case I just went back and tried it when the state changes to Historical and it does the same thing.

The reason I am even fooling with this is that I saw no way to set this in the Strategy Analyzer like you can on a Chart. The Set up section does not contain this setting. Perhaps you can enlighten me on why that is. Does Strategy Analyzer not support multiple Calculation modes? I sometimes change the Calculate mode to see what effect it has on the strategy outcome.

Finally, I am unsure of the terminology used in the quote above regarding a parent and child. Does this refer to inherited classes? My strategies all use a base class to handle functions common to all strategies and I handle mostly the entry logic in a class that inherits my base class (which inherits from the core Strategy class).

Thanks,

[NinjaTrader_Jesse]

Hello ChazJ,

This is not a setting which applies toward a backtest is the reason why its not present, your script is always calculated OnBarClose in historical. This is in the notes on the page you linked:

On a historical data set, only the OHLCVT of the bar is known and not each tick that made up the bar. As a result, State.Historical data processes OnBarUpdate() only on the close of each historical bar even if this property is set to OnEachTick or OnPriceChange.

The other note you are referring to would not apply in your described use case, that applies in a case where you develop your own indicator (hosted) and it needs to override the strategy (hosts) inherited calculate setting. In a strategy this should just be set from State.SetDefaults.


I look forward to being of further assistance.

[ChazJ]

Hi Jesse,

Thanks for all your help. Things are working smoothly now.

[ChazJ]

Hi Jesse,

Unfortunately this issue is back. I've been trying to narrow down the cause and it seems to be related to re-compiling while the Strategy Analyzer is active. Let me take you through the steps.

Using the SampleMACCrossOverTest strategy I uploaded for you, I added some code to keep track of things via the Output window. I added constructor code to the class where I assign a random Instance ID:

public SampleMACCrossOverTest()
{
mInstanceID = DateTime.Now.Ticks;
}

private long mInstanceID = 0;


Then I added a 'Print' statement when the State changes to Historical to track the Instance ID and the selected Average type:

else if (State == State.Historical)
{
Print("IsInstantiatedOnEachOptimizationIteration=" + IsInstantiatedOnEachOptimizationIteration.ToString () + ", InstanceID=" + mInstanceID.ToString() + ", AvgType=" + AvgType.ToString());
}

As you can see I am also printing the 'IsInstantiatedOnEachOptimizationIteration' variable because there is another issue going on with that which I will soon get to.

I compile this before opening the Strategy Analyzer.

Then I open Strategy Analyzer and run the strategy in Optimization mode with all twelve average types selected. Here are the results to the Output Window:

IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Exponential
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Hull
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054383041458, AvgType=TripleExponential
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Triangular
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Simple
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=DoubleExponential
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Kaufman
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054389883161, AvgType=Wilders
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054393254144, AvgType=Variable
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054394859850, AvgType=ZeroLagExponential
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054396076597, AvgType=VolumeWeighted
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054407236759, AvgType=Kaufman

All of the average types have been exercised, but you can see two issues:

1. There are 13 entries, not 12. Kaufman average type has been used twice. But, Strategy Analyzer only shows 12 in the results list even if I set the 'Keep best # results' to a higher number such as 15. It looks completely normal and correct.
2. Although the variable is set to 'true', you can see that Instance ID 637119054382682415 is re-used several times.

Now, I make an inconsequential change to the source by adding a blank line. I re-compile without closing the Strategy Analyzer window. and re-run the test. This is the Output Window:

IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616528094, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616498172, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616488200, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065625284673, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065626232142, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065626202220, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065628426555, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065639989964, AvgType=Triangular

Only two average types are exercised.

Finally, I re-run both tests above with the variable set to 'false'. I first close Strategy Analyzer, re-compile, then re-open SA to begin the sequence. Here is the first Output:

IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Simple
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Exponential
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=DoubleExponential
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273312364, AvgType=Kaufman
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273312364, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Hull
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273312364, AvgType=Triangular
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273312364, AvgType=TripleExponential
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Wilders
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Variable
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=VolumeWeighted
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=ZeroLagExponential
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068295833279, AvgType=Kaufman


After making another change to the source code and re-compiling without closing SA I get this:

IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070050065350, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070068895007, AvgType=TripleExponential

Same issue. But aside from the original issue, the variable 'IsInstantiatedOnEachOptimizationIteration' is not having the effect I would expect.

Obviously the simple workaround for the original issue is to close and restart SA after any and all re-compilations. But I hope you can fix the issue with this info because it is very time-consuming to do that on every source change.

Thanks

[NinjaTrader_Jesse]

Hello ChazJ,

Thank you for providing the extra details here, I was able to see the enum not iterating when doing the compile step. I will report this to development for further review.


Please let me know if I may be of further assistance.