How to Stop Backtest Iteration and Continue with Next

[tornadoatc]

Hello,

I have some two inputs that I want to optimize. I only want a backtest certain combinations of the two inputs. I want to know where I can evaluate the inputs and cancel the backtest based on certain combinations of the two inputs.

For instance, say first input is bool and second input is int and I only want to optimize the range of values of second input when the first input = true;

Is there a command to use inside the OnStateChange() method to abort the backtest iteration? Intent is to evaluate the two inputs and in certain combinations I would abort the backtest.

Thanks,

[NinjaTrader_Jesse]

Hello,

Thank you for the question.

This could be handled in a couple of ways, the way I would recommend would not be to abort the test but rather disable your logic if the property changes.

The other way would be using Disable() in NT7 or SetState(State.Terminated) in NT8. I did discover in NT8 this causes a crash in the beta and have submitted a request to development for that item so currently I would still only recommend using the first option.

I look forward to being of further assistance.

[tornadoatc]

Hello Jesse,

Has the issue with using SetState(State.Terminated) been resolved in NT8 Beta 8?

I would like to use this to Terminate execution of a given optimization run and have the strategy continue optimizing with the next set of variable inputs.

Thanks,

[NinjaTrader_Jesse]

Hello,

Thank you for the question.

I did test this on my end and no longer see the error, I wanted to check, from your own tests are you seeing the same?

If you have not yet updated to the current beta and tested this I would suggest doing that to confirm this has been resolved.

I look forward to being of further assistance.

[BarzTrading]

After a lot of testing, here's what I found worked in NT8 v8.1.1.3.

And by worked, I mean that I could terminate specific backtests during an optimization in Strategy Analyzer based on the value of a parameter, and the backtest would not appear in the final results. The optimization was significantly faster as a result of skipping those cases.

1)

First, call SetState when State=Historical.

if (State == State.Historical)
{
     if (Parm1 > 50)
     {
          SetState(State.Terminated);
          return;
     }
}
​

If you call before this state, SetState is ignored.

Also, I found that if I called SetState in OnBarUpdate as depicted on the help page, that it crashed the whole optimization.

2)

Next, if you want the performance benefit from skipping iterations, you need this:

IsInstantiatedOnEachOptimizationIteration = false

If this property is set to true, the backtest is still terminated and does not appear in the results, but for some reason, those terminated backtests still take just as long to process, if not a little longer.

---

Some of the current information on the SetState(...) page was a bit misleading

• The help page shows an example where SetState is called in OnBarUpdate. This might work in some scenarios, but not when optimizing in StrategyAnalyzer. It crashes the whole optimization.
• Also, the help page says "Warning: This method should only be call after the State reaches State.DataLoaded" The wording is ambiguous and suggests one could call SetState in the DataLoaded state. But it is only ignored here. This wording would be better: "This method should only be called after the DataLoaded state."

[NinjaTrader_ChelseaB]

Hello BarzTrading,

I've given this a test, and I am finding that calling SetState(State.Terminated) in State.DataLoaded is not causing a crash or any issues.

Further, if you are calling SetState(State.Terminated) in OnBarUpdate(), make sure you have a return under that line. Setting the state to terminated will not prevent the rest of the code in that OnBarUpdate() pass to stop evaluating. If there are orders placed, there won't be a strategy instance to place them with, and a null error will occur.

Below is a link to a video of the test.

https://drive.google.com/file/d/19RVQuuOVemOMRDue_W2k2zBgOW5XAV_F/view?usp=drivesdk

[BarzTrading]

Thank you, Chelsea.

I stand corrected on both counts. After re-testing this morning:

1) Yes, I can call SetState(State.Terminated) when State=State.DataLoaded and this works well.

As a side note, I often (but not always) lose print statements that were made right before the call to SetState in the DataLoaded state. I assume this is because the backtest is terminated before the statements get flushed to the console window. Not a problem, just an observation.

2) I was also able to call SetState(State.Terminated) in OnBarUpdate with no problems once I inserted a return so that no more code was executed after the call to SetState. This worked well and we still got the performance benefits, even when terminating in OnBarUpdate.

I really appreciate you helping to clarify my understanding further.

[NinjaTrader_ChelseaB]

Hello BarzTrading,

I would only expect "losing print statements" if there is a ClearOutputWindow() call somewhere.

[BarzTrading]

Thanks. There's no ClearOutputWindow() call.

Here's the console output. You can see that after the first few even numbered iterations, the print statements are being dropped. I don't know why.

Configure  : 11:43:00 AM, 0
Configure  : 11:43:00 AM, 2
Configure  : 11:43:00 AM, 1
Configure  : 11:43:00 AM, 5
Configure  : 11:43:00 AM, 3
Configure  : 11:43:00 AM, 4
Configure  : 11:43:00 AM, 6
DataLoaded : 11:43:00 AM, 6
DataLoaded : 11:43:00 AM, 1
DataLoaded : 11:43:00 AM, 3
DataLoaded : 11:43:00 AM, 5
DataLoaded : 11:43:00 AM, 4
DataLoaded : 11:43:00 AM, 2
DataLoaded : 11:43:00 AM, 4, State=Terminated
DataLoaded : 11:43:00 AM, 2, State=Terminated
DataLoaded : 11:43:00 AM, 6, State=Terminated
Historical : 11:43:01 AM, 5
Historical : 11:43:01 AM, 3
Historical : 11:43:01 AM, 1
Configure  : 11:43:01 AM, 11
DataLoaded : 11:43:01 AM, 11
Historical : 11:43:01 AM, 11
Configure  : 11:43:01 AM, 9
DataLoaded : 11:43:01 AM, 9
Historical : 11:43:01 AM, 9
Configure  : 11:43:01 AM, 7
DataLoaded : 11:43:01 AM, 7
Historical : 11:43:01 AM, 7
Configure  : 11:43:01 AM, 17
DataLoaded : 11:43:01 AM, 17
Historical : 11:43:01 AM, 17
Configure  : 11:43:01 AM, 15
DataLoaded : 11:43:01 AM, 15
Historical : 11:43:01 AM, 15
Configure  : 11:43:01 AM, 13
DataLoaded : 11:43:01 AM, 13
Historical : 11:43:01 AM, 13
Configure  : 11:43:01 AM, 19
DataLoaded : 11:43:01 AM, 19
Historical : 11:43:01 AM, 19
Configure  : 11:43:02 AM, 5
DataLoaded : 11:43:02 AM, 5
Historical : 11:43:02 AM, 5

​​

Here's the strategy:

using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using NinjaTrader.Cbi;

namespace NinjaTrader.NinjaScript.Strategies.Func
{
    /// <summary>
    /// https://forum.ninjatrader.com/forum/ninjatrader-7/general-development/82083-how-to-stop-backtest-iteration-and-continue-with-next
    /// </summary>
    public class Func_IterationTerminate : Strategy
    {

        [NinjaScriptProperty]
        [Display(Name = "Parameter 1", Description = "", GroupName = "Parameters", Order = 1)]
        public int Parm1 { get; set; }

        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Name        = "Func - Iteration Terminate";
                IsInstantiatedOnEachOptimizationIteration = false;
            }
            else if (State == State.Configure)
            {
                string msg = String.Format("Configure  : {0}, {1}", DateTime.Now.ToLongTimeString(), Parm1);
                Print(msg);
            }
            else if (State == State.DataLoaded)
            {
                string msg = String.Format("DataLoaded : {0}, {1}", DateTime.Now.ToLongTimeString(), Parm1);
                Print(msg);

                if (Parm1 % 2 == 0)
                {
                    msg = String.Format("DataLoaded : {0}, {1}, State=Terminated", DateTime.Now.ToLongTimeString(), Parm1);
                    Print(msg);
                    SetState(State.Terminated);
                    return;
                }
            }
            else if (State == State.Historical)
            {
                string msg = String.Format("Historical : {0}, {1}", DateTime.Now.ToLongTimeString(), Parm1);
                Print(msg);
            }
        }

        protected override void OnBarUpdate()
        {
            if (CurrentBar < BarsRequiredToTrade)
            {
                return;
            }

            if (Position.MarketPosition == MarketPosition.Flat)
            {
                EnterLong();
            }
            else if (BarsSinceEntryExecution() > 10)
            {
                ExitLong();
            }

        }

    }
}​

Here's StrategyAnalyzer:

[NinjaTrader_ChelseaB]

Hello BarzTrading,

I'm not clearly seeing the issue.

Which print is not appearing that you are expecting?

[BarzTrading]

In the console output, we have lines for all odd parameter values, as expected. One line each for Configure, DataLoaded, and Historical. We have some lines for parameter values 2, 4 and 6: Configure, DataLoaded, and State=Terminated.

But no line are printed for parameter values 8, 10, 12, 14, 16, 18, 20.

We see the line with "State=Terminated" for only 3 cases, but in fact, 10 cases are terminated. Why are three cases printed, but not the other 7?

[NinjaTrader_ChelseaB]

Hello BarzTrading,

Thank you for clarifying.

If you comment out the SetState() line in State.DataLoaded you are seeing these instances print?
If so, it may be that terminating the script is happening too quickly and nulling the instance before the print can go through.

[BarzTrading]

Thanks. Yes, if I comment SetState, I see all the prints.

I also see all of the prints if I set IsInstantiatedOnEachOptimizationIteration=true.

I think it's possibly a timing issue where the strategy instances are terminated and re-used so quickly that it prevents the print.

The first three prints ALWAYS show up on the console for the first three terminated backtests. I suspect this has something to do with the fact that this is the first use of those three strategy instances. There are six strategy instances created, I assume for six threads. Three handle odd instances and three handle even instances. So it's probably just that the prints make it through on the first use of any given instance.

This is the console output when IsInstantiatedOnEachOptimizationIteration=true. We see all print statements:

Configure : 4:28:39 PM, 0, 613d2d99
Configure : 4:28:40 PM, 1, 33ca9a92
Configure : 4:28:40 PM, 5, 376586b7
Configure : 4:28:40 PM, 2, 4a50e19c
Configure : 4:28:40 PM, 3, b3f2b463
Configure : 4:28:40 PM, 6, b0b73160
Configure : 4:28:40 PM, 4, 92291d18
DataLoaded : 4:28:40 PM, 2, 4a50e19c
DataLoaded : 4:28:40 PM, 6, b0b73160
DataLoaded : 4:28:40 PM, 1, 33ca9a92
DataLoaded : 4:28:40 PM, 2, 4a50e19c, State=Terminated
DataLoaded : 4:28:40 PM, 6, b0b73160, State=Terminated
DataLoaded : 4:28:40 PM, 3, b3f2b463
DataLoaded : 4:28:40 PM, 5, 376586b7
DataLoaded : 4:28:40 PM, 4, 92291d18
DataLoaded : 4:28:40 PM, 4, 92291d18, State=Terminated
Historical : 4:28:40 PM, 3, b3f2b463
Historical : 4:28:40 PM, 5, 376586b7
Configure : 4:28:40 PM, 10, d18c19df
DataLoaded : 4:28:40 PM, 10, d18c19df
DataLoaded : 4:28:40 PM, 10, d18c19df, State=Terminated
Configure : 4:28:40 PM, 8, fd5fd7af
DataLoaded : 4:28:40 PM, 8, fd5fd7af
DataLoaded : 4:28:40 PM, 8, fd5fd7af, State=Terminated
Configure : 4:28:40 PM, 12, bbca3764
DataLoaded : 4:28:40 PM, 12, bbca3764
DataLoaded : 4:28:40 PM, 12, bbca3764, State=Terminated
Historical : 4:28:40 PM, 1, 33ca9a92
Configure : 4:28:40 PM, 9, aac62e32
DataLoaded : 4:28:40 PM, 9, aac62e32
Configure : 4:28:40 PM, 11, 44211212
DataLoaded : 4:28:40 PM, 11, 44211212
Configure : 4:28:41 PM, 16, cfc09d52
DataLoaded : 4:28:41 PM, 16, cfc09d52
DataLoaded : 4:28:41 PM, 16, cfc09d52, State=Terminated
Configure : 4:28:41 PM, 14, 847655f4
DataLoaded : 4:28:41 PM, 14, 847655f4
DataLoaded : 4:28:41 PM, 14, 847655f4, State=Terminated
Configure : 4:28:41 PM, 7, 5f5c005e
DataLoaded : 4:28:41 PM, 7, 5f5c005e
Configure : 4:28:41 PM, 18, e4466db7
DataLoaded : 4:28:41 PM, 18, e4466db7
DataLoaded : 4:28:41 PM, 18, e4466db7, State=Terminated
Historical : 4:28:41 PM, 9, aac62e32
Historical : 4:28:41 PM, 11, 44211212
Configure : 4:28:41 PM, 15, 84ee93f5
DataLoaded : 4:28:41 PM, 15, 84ee93f5
Configure : 4:28:41 PM, 20, 5fae6b9e
DataLoaded : 4:28:41 PM, 20, 5fae6b9e
DataLoaded : 4:28:41 PM, 20, 5fae6b9e, State=Terminated
Historical : 4:28:41 PM, 7, 5f5c005e
Configure : 4:28:41 PM, 17, 3056c190
DataLoaded : 4:28:41 PM, 17, 3056c190
Configure : 4:28:41 PM, 13, 2b097726
DataLoaded : 4:28:41 PM, 13, 2b097726
Historical : 4:28:41 PM, 15, 84ee93f5
Historical : 4:28:42 PM, 17, 3056c190
Historical : 4:28:42 PM, 13, 2b097726
Configure : 4:28:42 PM, 19, c1041d6f
DataLoaded : 4:28:42 PM, 19, c1041d6f
Historical : 4:28:42 PM, 19, c1041d6f
Configure : 4:28:43 PM, 3, 90940ad5
DataLoaded : 4:28:43 PM, 3, 90940ad5
Historical : 4:28:43 PM, 3, 90940ad5​

This is the console output when IsInstantiatedOnEachOptimizationIteration=false.

Configure : 4:38:14 PM, 0, dbdb9e99
Configure : 4:38:15 PM, 5, 552c7817
Configure : 4:38:15 PM, 3, 6101c0e8
Configure : 4:38:15 PM, 2, 386fd739
Configure : 4:38:15 PM, 1, 9a070f6d
Configure : 4:38:15 PM, 6, 0117429b
Configure : 4:38:15 PM, 4, caea31d6
DataLoaded : 4:38:15 PM, 4, caea31d6
DataLoaded : 4:38:15 PM, 3, 6101c0e8
DataLoaded : 4:38:15 PM, 6, 0117429b
DataLoaded : 4:38:15 PM, 4, caea31d6, State=Terminated
DataLoaded : 4:38:15 PM, 5, 552c7817
DataLoaded : 4:38:15 PM, 6, 0117429b, State=Terminated
DataLoaded : 4:38:15 PM, 1, 9a070f6d
DataLoaded : 4:38:15 PM, 2, 386fd739
DataLoaded : 4:38:15 PM, 2, 386fd739, State=Terminated
Historical : 4:38:15 PM, 3, 6101c0e8
Historical : 4:38:15 PM, 5, 552c7817
Historical : 4:38:15 PM, 1, 9a070f6d
Configure : 4:38:16 PM, 9, 6101c0e8
DataLoaded : 4:38:16 PM, 9, 6101c0e8
Historical : 4:38:16 PM, 9, 6101c0e8
Configure : 4:38:16 PM, 11, 552c7817
DataLoaded : 4:38:16 PM, 11, 552c7817
Historical : 4:38:16 PM, 11, 552c7817
Configure : 4:38:16 PM, 7, 9a070f6d
DataLoaded : 4:38:16 PM, 7, 9a070f6d
Historical : 4:38:16 PM, 7, 9a070f6d
Configure : 4:38:16 PM, 15, 6101c0e8
DataLoaded : 4:38:16 PM, 15, 6101c0e8
Historical : 4:38:16 PM, 15, 6101c0e8
Configure : 4:38:16 PM, 17, 552c7817
DataLoaded : 4:38:16 PM, 17, 552c7817
Historical : 4:38:16 PM, 17, 552c7817
Configure : 4:38:16 PM, 13, 9a070f6d
DataLoaded : 4:38:16 PM, 13, 9a070f6d
Historical : 4:38:16 PM, 13, 9a070f6d
Configure : 4:38:16 PM, 19, 9a070f6d
DataLoaded : 4:38:16 PM, 19, 9a070f6d
Historical : 4:38:16 PM, 19, 9a070f6d
Configure : 4:38:17 PM, 3, 3972f5dc
DataLoaded : 4:38:17 PM, 3, 3972f5dc
Historical : 4:38:17 PM, 3, 3972f5dc​

Anyway, it's only a curiosity at this point. I need to move on to my next project.

Thanks for the assistance.

[NinjaTrader_ChelseaB]

Hello Duane95,

NinjaTrader is on release 8.1.1.7 and has been out of beta for quite some time.

However, if a script is terminated its not able to continue printing or taking actions.

If you need further actions, I would recommend returning in OnBarUpdate() and allowing the strategy to complete the backtest without evaluating any logic (so it will be quick).

[madb123]

Thanks for the above, this thread has improved my optimisation speed signifficantly.
I'm trying to also use it for genetic optimisation but it seems to crashes.

Is this expected because i am terminating offspring from each generation?
is there a way to terminate/ take out useless iterations from a genetic optimisation run?