Walk Forward Optimization Issues

[linuxguru]

I see great potential with the walk-forward optimization, but there are a few issues with it.

1. I want to be able to start the testing period on a week or month boundary. What value for "test period (days)" can I use to make it start every Sunday (for example)?

2. It appears that when it is running the test period, "bars required to trade" starts at the beginning of the test period. It would make more sense to start this before the test period so the indicators are preloaded with history going into the test period. Additionally, some indicators don't start operating until that number of bars is passed (SMA for example) so at the beginning of the test period, after the bars required to trade period, the indicator is just broken (see attached screenshot of a simple moving average on a 15 min bat chart).

If there is anyway to solve these problems, please let me know. I don't see any source code that would allow me to change any of the behavior I have mentioned, but if there is, please point me in the right direction.

Otherwise, please consider adding them to the list of things to implement in future versions of NinjaTrader.

Thanks,
John

[NinjaTrader_Jesse]

Hello linuxguru,

Thank you for the post.

There is not currently a way to specify a testing day in the walk forward directly. This is something you can control in your strategies logic to only trade on those days but that does not change how the walk forward will gather its data or process. You could try to use the values in the walk forward to make a period of a week using the end date to do that but if there are any holidays or any changes in the data that won't continue to land on the same day.

The Bars required to trade and plot can be set from code to 0 but would not be able to start prior to the test period as you asked. I believe for both of these items I would need to put in a feature request.
In the case of a system indicator you would have to duplicate it and add the BarsRequiredToPlot = 0 to make it start processing immediately.

https://ninjatrader.com/support/helpGuides/nt8/barsrequiredtotrade.htm?zoom_highlightsub=barsrequired

https://ninjatrader.com/support/helpGuides/nt8/barsrequiredtoplot.htm?zoom_highlightsub=barsrequired

I look forward to being of further assistance.

[linuxguru]

Yes, submit these as feature requests. #2 is far more important than #1 if you want to prioritize them. Here's why:

IMO it is unacceptable to include the bars required to trade in the test period. Many (if not all) indicators require several bars before they can calculate anything (if they rely on historical bars), and then several more bars before they produce accurate results (such as the moving average with a period larger than 0, for example). In a real trading scenario, there would be at a minimum of 3 days of intraday bars before the indicators would be used to make trading decisions. Having the optimization process use the indicators for decisions with no history before the test period makes the walk-forward option highly unrealistic, and therefore nearly useless for testing, especially when the testing period is short.

Are you telling me that setting BarsRequiredToPlot = 0 will cause a moving average to look back "period" number of bars before the first bar and calculate the moving average based on previous history? I believe the chart shows this is not the case. If not, then what is the point in allowing a moving average to plot that hasn't seen "period" number of bars yet? It can't possibly plot anything useful. It is obvious from the chart I provided that the moving average was fed zeros prior to it plotting, and then more zeros until "bars required to trade" bars had past (100 in this case). After that, the indicator ramped up to the actual price as the sum of many of zeros and a little data increased to all data and no zeros. Any trading decision based on that indicator would be a bad one.

The purpose of the testing period is to actually trade during that period. I believe the way to fix this is to feed the strategy from the prior optimization bars with the settings that are determined to be optimal for that period as history into the testing period, and then run the testing period with the same settings, but only record the trades that occur during the testing period.

I look forward to seeing this fixed properly in a future release.

[NinjaTrader_Jesse]

Are you telling me that setting BarsRequiredToPlot = 0 will cause a moving average to look back "period" number of bars before the first bar and calculate the moving average based on previous history?

No, this just allows for processing before 20 bars or on bar 0. If the script has logic to account for a variable period it will likely make use of that here, if the script has hard coded periods or values it may have difficulty with that change and not having enough data. The SMA has one example of that type of logic with the period:

CurrentBar < Period ? CurrentBar + 1 : Period

Either the period is used or CurrentBar+1 depending on if the current bar is less than the period, or bar 1 with a 20 period for example.

If not, then what is the point in allowing a moving average to plot that hasn't seen "period" number of bars yet. It can't possibly plot anything useful. It is obvious from the chart I provided that the moving average was fed zeros prior to it plotting, and then more zeros until "bars required to trade" bars had past (100 in this case). After that, the indicator ramped up to the actual price as the sum of many of zeros and a little data increased to all data and no zeros. Any trading decision based on that indicator would be a bad one.

That really depends on the indicator, in your example that is not likely useful data. Other indicators may require starting from bar 0 or doing calculations early to be accurate and may set this property to 0 or a really high number. This is just one of many features of NinjaScript that allows for flexible calculation. One use case where this comes in handy is for a poorly programmed indicator that has no error checking for data requirements. Applying a default 20 bars may allow that indicator to work when it otherwise should not based on how it was programmed.

I look forward to being of further assistance.

[NT-Roland]

Hi Jesse,
This comes down to an old request of mine called "pre-load bars".

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/1035778-preload-bars

Very often, you have historical data for a much longer period than you want to run backtests for.
I don't get it why it should not be possible to call historical data for an additional period and start trading IMMEDIATELY with all indicators prepopulated as needed.
So stupid. Competition can handle this most basic functionality. Sorry for the tone, but look at the date of my request. Not making ANY progress.
NT-Roland

[linuxguru]

I agree. However, the issue is amplified with walk-forward because the testing period is reduced. I love a lot about NinjaTrader (I invested in the software) and I'd love even more for some of these fairly small issues to be resolved.

Thanks,
John

[MojoJojo]

Hi,

walk forward testing is a great idea, however only testing the best result after the optimization period is a bit restrictive and not as productive as it could be in my opinion.
Currently "Keep best # results" only effect is that the optimization results are kept for analysis.

First eliminating bad parameters and then testing will in many cases eliminate bad combinations and save time compared to a full optimization.
It's very likely that during the testing period other combinations will outperform the only result NT chooses for testing right now.

Therefore my request in addition to linuxguru's and NT-Roland's is

3) Keep best # results or additional parameter should change how many test iterations are run

[MojoJojo]

Never mind, it's possible to implement an elimination algorithm with the regular optimizer.
My implementation isn't able to have a fixed number of strategies survive but it will weed out the weak.
The example removes iterations that were not profitable or didn't trade enough every 100 days.
The optimization finished 30% faster with 90% fewer results, bad strategies and more stringent criteria speed up the process.

If anybody wants to try this, do so at your own risk, also the optimizer will not display any iteration that were terminated manually.
I haven't tested the code with anything other than the regular optimizer, walk foward etc. could be negatively affected by it.

        private double optimizationProfit = 0;
        private DateTime optimizationLastMetricsCheckDate = DateTime.MinValue;
        private int optimizationTradeCount = 0;

...
        private bool IsOptimization() {
            return Category == Category.Optimize || Category == Category.MultiObjective || Category == Category.WalkForward;
        }

        protected bool MeetsPerformanceRequirements() {
            /* //instead of terminating strategies can be stopped from processing bars
               //this will save less performance but undesirable results will be included in optimizer output
            if(!meetsPerformanceRequirements) {
                return false;
            }

            if(!IsOptimization() && !(Category == Category.Backtest && Optimizer != null)) { //optimizer  re runs results with category backtest
                return true;
            }
            */

            if(!IsOptimization()) { //use other IsOptimizer check if undesirable results should be included
                return true;
            }

            if(optimizationLastMetricsCheckDate == DateTime.MinValue) {
                optimizationLastMetricsCheckDate = Time[0].AddDays(OptimizationProfitabilityCheckDays);
                return true;
            }

            if(optimizationLastMetricsCheckDate.CompareTo(Time[0]) > 0) {
                return true;
            }

            int tradeCount = SystemPerformance.AllTrades.Count;
            double profit = SystemPerformance.AllTrades.Sum(v => v.ProfitCurrency);
            if(tradeCount - optimizationTradeCount < OptimizationMinimumTradesPerInterval
                || profit - optimizationProfit <= 0) {
                SetState(State.Terminated);
                meetsPerformanceRequirements = false;
                return false;
            }
            optimizationProfit = profit;
            optimizationTradeCount = tradeCount;
            optimizationLastMetricsCheckDate = Time[0].AddDays(OptimizationProfitabilityCheckDays);
            return true;                
        }
...
        protected override void OnBarUpdate() {
            if(BarsInProgress != 0) {//optional
                return;
            }

            if(!MeetsPerformanceRequirements()) {
                return;
            }
...
        [NinjaScriptProperty]
        [Display(Name="Optimization Profitability Check Days", Order=997, GroupName="Parameters")]
        [Range(0, int.MaxValue)]
        public int OptimizationProfitabilityCheckDays
        { get; set; }

        [NinjaScriptProperty]
        [Display(Name="Optimization Minimum Trades", Order=998, GroupName="Parameters")]
        [Range(0, int.MaxValue)]
        public int OptimizationMinimumTradesPerInterval
        { get; set; }