Walk Forward Test Crashes

[bkonia]

I'm noticing some strange crashing behavior when performing walk forward tests on Forex instruments in NT 8.0.22.2 (64-bit). Consider the example strategy below.

if you run this strategy as a Walk Forward Test from 2010 - present, on any Forex instrument, using an OptimizationPeriod greater than 30, the walk forward test crashes on the first iteration and returns no results. The crashing seems to be related to the fact that the strategy has no exits.

• If you add an exit condition, it does not crash.
• If you use an OptimizationPeriod of 30 or less, it does not crash.
• If you test against a non-Forex instrument it does not crash.

I realize it may seem strange to create a strategy that has no exit conditions, but I did this on purpose to simulate crashing in a real strategy. I noticed when running a basket walk forward test against a group of instruments, the test would crash on any instrument that had no exits during an optimization period. The problem is, it crashes the entire optimization, not just the affected instrument.

I know you're going to ask for log and trace files, but I already checked them and there's nothing unusual; no reported errors or crashes in either file. To reproduce the issue, please run my test strategy against a Forex instrument with an OptimizationPeriod > 30.

namespace NinjaTrader.NinjaScript.Strategies
{
    public class Test : Strategy
    {
        protected override void OnStateChange()
        {
            if (State == State.SetDefaults)
            {
                Description                                    = @"Enter the description for your new custom Strategy here.";
                Name                                        = "Test";
                Calculate                                    = Calculate.OnBarClose;
                EntriesPerDirection                            = 1;
                EntryHandling                                = EntryHandling.AllEntries;
                IsExitOnSessionCloseStrategy                = true;
                ExitOnSessionCloseSeconds                    = 30;
                IsFillLimitOnTouch                            = false;
                MaximumBarsLookBack                            = MaximumBarsLookBack.TwoHundredFiftySix;
                OrderFillResolution                            = OrderFillResolution.Standard;
                Slippage                                    = 0;
                StartBehavior                                = StartBehavior.WaitUntilFlat;
                TimeInForce                                    = TimeInForce.Gtc;
                TraceOrders                                    = false;
                RealtimeErrorHandling                        = RealtimeErrorHandling.StopCancelClose;
                StopTargetHandling                            = StopTargetHandling.PerEntryExecution;
                BarsRequiredToTrade                            = 20;
                // Disable this property for performance gains in Strategy Analyzer optimizations
                // See the Help Guide for additional information
                IsInstantiatedOnEachOptimizationIteration    = true;
                MyParameter                    = 1;
            }
            else if (State == State.Configure)
            {
            }
        }

        protected override void OnBarUpdate()
        {
            EnterLong();
        }

        #region Properties
        [NinjaScriptProperty]
        [Range(1, int.MaxValue)]
        [Display(Name="MyParameter", Order=1, GroupName="Parameters")]
        public int MyParameter
        { get; set; }
        #endregion

    }
}

[NinjaTrader_Kate]

Hello bkonia,

Thank you for your note.

I'm not seeing any crashing when I use this to perform a walk forward with an optimization period of over 30 with a forex instrument (I used USDJPY since I had that handy). What version of NinjaTrader are you currently running? You can find this under Help > About. The current version is 8.0.22.2.

Also, what interval are you using? I used a 1 minute chart.

Thanks in advance; I look forward to assisting you further.

[bkonia]

Hi Kate,

As I mentioned in my original post, the version is 8.0.22,2. Try it with the following settings:

Instrument: GBPSEK
Interval: Daily
Optimization Period: 31
Start Date: 1/1/2010
End Date: 6/14/2020
Optimize on: Max strength
Optimizer: Default

[NinjaTrader_Kate]

Hello bkonia,

Thank you for your reply.

I'm still not seeing crashing with this - simply that it ends with no results because no trade is ever completed.

I'm unable to test with that particular forex pair as our test connection doesn't provide data for it, but all the other pairs I've tried have had the same result - it stops after 1 iteration and gives 0 results, as I would expect from a strategy that never completes a trade.

Let's have you try clearing the cache and see if perhaps there's a data issue that may be causing the crashing.

To delete the cache, follow the steps below.

• Shut down NinjaTrader.
• Open the Documents > NinjaTrader 8 > db folder.
• Delete the sub-folder named 'cache'.
• Restart NinjaTrader and test.

Please let us know if we may be of further assistance.

[bkonia]

Hi Kate,

The behavior that you're experiencing (stopping after 1 iteration) is the same thing that I'm seeing and it's incorrect.

The fact that it has no trades on the first iteration does not justify stopping the test. The whole point of running an optimization is to test different parameter combinations. Normally, each iteration will return a different number of trades and depending on your parameter settings, it's certainly possible that some iterations will have no trades. In this example, it's stopping the optimization after 1 iteration, but NinjaTrader has no way of knowing that subsequent iterations will also have no trades.

Furthermore, if you run the optimization on multiple instruments (basket test), it stops the entire optimization for all instruments after 1 iteration on 1 instrument. This is obviously incorrect behavior and even though it's not displaying an error message, I have to assume it's crashing.

[NinjaTrader_Kate]

Hello bkonia,

Thank you for your reply.

I will reach out to our development team on this and let you know their findings - this wouldn't be a crash per se, as the platform still functions for me when this occurs, so I'll double check with them to see if this would be the expected behavior.

Thanks in advance; I look forward to assisting you further.

[bkonia]

Hi Kate,

Thank you for following up on this. I understand it's not technically speaking a "crash", but since the optimization aborts prematurely, the effect is the same. I don't see how this could possibly be expected behavior. It makes no sense to stop the optimization after 1 iteration, since with different parameter settings, the strategy may very well have trades. Prior to trying each optimization iteration, NT has no way of knowing whether or not there will be trades during that iteration.

Finally, as further evidence that this is incorrect behavior, if you change the optimization target to something other than Max Strength (for example, Max Profit Factor), it runs through all the iterations without aborting. Therefore, the issue seems to be specific to using Max Strength as your optimization target.

[NinjaTrader_Kate]

Hello bkonia,

Thank you for your patience.

I just wanted to update you that I've sent this through to our development team and am waiting for their response on this issue. Hopefully I will have some further information for you soon.

Thanks in advance; I look forward to assisting you further.

[NinjaTrader_Kate]

Hello bkonia,

Thank you for your patience.

Our development team has let me know they have a build they'd like you to test that should resolve this. Please write into us at platformsupport [at] ninjatrader [dot] com directly so we can provide you with instructions and a download link. Please include 2625076 Attn Kate W. in the Subject line and a link to this thread in the body of the email.

Thanks in advance; I look forward to assisting you further.