• If this is your first visit, you will have to register before you can post. To view messages, please scroll below and select the forum that you would like to visits. Questions? Be sure to check out the Forum FAQ.

Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Strategy Analyzer Stops Iterating Parameter Selections During Optimization

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Strategy Analyzer Stops Iterating Parameter Selections During Optimization

    As I run the Strategy Analyzer in optimization mode with various parameter settings I have noticed that it will get into a state where the selected parameters are not being exercised.

    As an example, I have a strategy which uses moving averages and I have a parameter to select the type of moving average so I can see which one works best. When I first run it, it works fine. However, later, after playing around with some other parameters, I can see that the Analyzer is no longer going through the list of moving average types. It is always selecting the default. The results tab shows that the settings are different for each run, but the results are identical. This is what led me to question the operation. I then verified this using Visual Studio and setting break points as it is running. It always uses the default as it is running.

    If I close Strategy Analyzer and re-open it, proper operation is resumed until some later point in my testing. I recently upgraded to build 20. I am not sure if this was happening in build 19. I have to assume it was. So, obviously, I have to now invalidate all my previous results of testing, which has cost me a lot of wasted time.

    Please let me know what other info I can supply you with to troubleshoot this issue.

    Thanks,

    #2
    Hello ChazJ,

    Thank you for the post.

    I was unable to locate any current issues which match this description, we would likely need to see an example of the syntax used along with the settings used in the optimization when you see this to see if this can be replicated.

    For testing I would suggest using the SampleMACrossOver to see if you are able to replicate it with that logic. If the sample cannot reproduce the problem this may relate to something specific being done in the custom strategies logic.

    Can you additionally check the log tab when you see this happen to check if any errors are being reported?

    I look forward to being of further assistance.




    JesseNinjaTrader Customer Service

    Comment


      #3
      Hi Jesse,

      I think I have found a way to reproduce this issue with the SampleMACCrossOver strategy, but I had to modify it to add the parameter selection for Average type. I have attached the modified sample.

      The item that causes the issue to occur is the line:

      // Calculate = Calculate.OnBarClose;

      in the OnStateChange method under State == State.DataLoaded

      Run this sample, then remove the comments from that statement and run it again. The Debug.Print statement will show that the same average type is being used for each instance of the optimization run. The strange thing is that the Calculate value is already set to OnBarClose, so this statement should be having no effect. Nevertheless there must be some code behind the property setting that is causing this issue.

      I hope you are able to reproduce it using this code.

      Regards,
      Attached Files

      Comment


        #4
        Hello ChazJ,

        Is there a reason this is being set from DataLoaded and not SetDefaults? This is not expected to ever be toggled later in logic so I would not suspect this would be a problem just wrong placement of where the property being set.

        If you instead use State.SetDefaults do your tests succeed?


        I look forward to being of further assistance.
        JesseNinjaTrader Customer Service

        Comment


          #5
          Hi Jesse,

          I was prompted by the Help Documentation located here: https://ninjatrader.com/support/help.../calculate.htm

          where it states:

          "Typically this property would be set in State.SetDefaults, however scripts that require Calculate to be set by the developer must set this property in State.Historical in its OnStateChange() in order to ensure that if this script is a child (hosted) that the parent.Calculate property which is adopted by the child is overridden again."

          I had moved it from State.Historical to State.DataLoaded figuring it would be better to set it earlier. In any case I just went back and tried it when the state changes to Historical and it does the same thing.

          The reason I am even fooling with this is that I saw no way to set this in the Strategy Analyzer like you can on a Chart. The Set up section does not contain this setting. Perhaps you can enlighten me on why that is. Does Strategy Analyzer not support multiple Calculation modes? I sometimes change the Calculate mode to see what effect it has on the strategy outcome.

          Finally, I am unsure of the terminology used in the quote above regarding a parent and child. Does this refer to inherited classes? My strategies all use a base class to handle functions common to all strategies and I handle mostly the entry logic in a class that inherits my base class (which inherits from the core Strategy class).

          Thanks,

          Comment


            #6
            Hello ChazJ,

            This is not a setting which applies toward a backtest is the reason why its not present, your script is always calculated OnBarClose in historical. This is in the notes on the page you linked:
            On a historical data set, only the OHLCVT of the bar is known and not each tick that made up the bar. As a result, State.Historical data processes OnBarUpdate() only on the close of each historical bar even if this property is set to OnEachTick or OnPriceChange.
            The other note you are referring to would not apply in your described use case, that applies in a case where you develop your own indicator (hosted) and it needs to override the strategy (hosts) inherited calculate setting. In a strategy this should just be set from State.SetDefaults.


            I look forward to being of further assistance.
            JesseNinjaTrader Customer Service

            Comment


              #7
              Hi Jesse,

              Thanks for all your help. Things are working smoothly now.

              Comment


                #8
                Hi Jesse,

                Unfortunately this issue is back. I've been trying to narrow down the cause and it seems to be related to re-compiling while the Strategy Analyzer is active. Let me take you through the steps.

                Using the SampleMACCrossOverTest strategy I uploaded for you, I added some code to keep track of things via the Output window. I added constructor code to the class where I assign a random Instance ID:

                public SampleMACCrossOverTest()
                {
                mInstanceID = DateTime.Now.Ticks;
                }

                private long mInstanceID = 0;


                Then I added a 'Print' statement when the State changes to Historical to track the Instance ID and the selected Average type:

                else if (State == State.Historical)
                {
                Print("IsInstantiatedOnEachOptimizationIteration=" + IsInstantiatedOnEachOptimizationIteration.ToString () + ", InstanceID=" + mInstanceID.ToString() + ", AvgType=" + AvgType.ToString());
                }

                As you can see I am also printing the 'IsInstantiatedOnEachOptimizationIteration' variable because there is another issue going on with that which I will soon get to.

                I compile this before opening the Strategy Analyzer.

                Then I open Strategy Analyzer and run the strategy in Optimization mode with all twelve average types selected. Here are the results to the Output Window:

                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Exponential
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Hull
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054383041458, AvgType=TripleExponential
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Triangular
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Simple
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=DoubleExponential
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054382682415, AvgType=Kaufman
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054389883161, AvgType=Wilders
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054393254144, AvgType=Variable
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054394859850, AvgType=ZeroLagExponential
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054396076597, AvgType=VolumeWeighted
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119054407236759, AvgType=Kaufman

                All of the average types have been exercised, but you can see two issues:

                1. There are 13 entries, not 12. Kaufman average type has been used twice. But, Strategy Analyzer only shows 12 in the results list even if I set the 'Keep best # results' to a higher number such as 15. It looks completely normal and correct.
                2. Although the variable is set to 'true', you can see that Instance ID 637119054382682415 is re-used several times.

                Now, I make an inconsequential change to the source by adding a blank line. I re-compile without closing the Strategy Analyzer window. and re-run the test. This is the Output Window:

                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616528094, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616498172, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616488200, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065616189011, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065625284673, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065626232142, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065626202220, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065628426555, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=True, InstanceID=637119065639989964, AvgType=Triangular

                Only two average types are exercised.

                Finally, I re-run both tests above with the variable set to 'false'. I first close Strategy Analyzer, re-compile, then re-open SA to begin the sequence. Here is the first Output:

                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Simple
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Exponential
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=DoubleExponential
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273312364, AvgType=Kaufman
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273312364, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Hull
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273312364, AvgType=Triangular
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273312364, AvgType=TripleExponential
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Wilders
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=Variable
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=VolumeWeighted
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068273292399, AvgType=ZeroLagExponential
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119068295833279, AvgType=Kaufman


                After making another change to the source code and re-compiling without closing SA I get this:

                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070050065350, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070049686501, AvgType=MESAAdaptive
                IsInstantiatedOnEachOptimizationIteration=False, InstanceID=637119070068895007, AvgType=TripleExponential

                Same issue. But aside from the original issue, the variable 'IsInstantiatedOnEachOptimizationIteration' is not having the effect I would expect.

                Obviously the simple workaround for the original issue is to close and restart SA after any and all re-compilations. But I hope you can fix the issue with this info because it is very time-consuming to do that on every source change.

                Thanks

                Comment


                  #9
                  Hello ChazJ,

                  Thank you for providing the extra details here, I was able to see the enum not iterating when doing the compile step. I will report this to development for further review.


                  Please let me know if I may be of further assistance.
                  JesseNinjaTrader Customer Service

                  Comment

                  Latest Posts

                  Collapse

                  Topics Statistics Last Post
                  Started by itrader46, Today, 02:49 PM
                  2 responses
                  6 views
                  0 likes
                  Last Post itrader46  
                  Started by m94306, Today, 03:01 PM
                  1 response
                  6 views
                  0 likes
                  Last Post NinjaTrader_Heath  
                  Started by fx.practic, Today, 02:54 PM
                  3 responses
                  8 views
                  0 likes
                  Last Post fx.practic  
                  Started by MarianApalaghiei, 07-14-2019, 04:00 AM
                  5 responses
                  62 views
                  0 likes
                  Last Post koganam
                  by koganam
                   
                  Started by pstrusi, Today, 02:05 PM
                  2 responses
                  7 views
                  0 likes
                  Last Post pstrusi
                  by pstrusi
                   
                  Working...
                  X