Why you should not trust walk-forward results...

[koganam]

...
So in my case, I had BRTT=14, SMA period 10. Each test period will therefore ignore the first 14 bars, then calculate SMA(10) with all valid data.
There is NO reference to bars prior to the start of the SMA period.


[ATTACH]n1070482[/ATTACH]

Cheers,
T.

Hm. I would then argue that: "...Each test period will therefore ignore the first 14 bars" is something that the backtest does not do. So if, in the backtest, the crossover happened in those 14 bars, then that crossover will not show up in the WFO. So regardless, we still have SMA that differ in behavior from those in the backtest. Indeed, it might actually be better if the SMA actually did use the data from previous periods, thus retaining a consistent SMA between all period slices and the backtest.

[tgn55]

Yep.. for sure if the crossover happens in one of the WF "blind spots", the strategy will miss it, agreed. Just one of the inevitable consequences of WF test period starting over. The simple backtest ignores the first 14 bars ... and then it's on with the show. WF repeats that for each and every test period. In fact, this may well explain at least some of the huge variations I saw in my strategy's WF when I changed the start date... only a bit... but who knows what trades it missed... or took ... because of that.

But I don't agree with your comment re SMA. The previous post proved conclusively that SMA is indeed consistent... and also implemented precisely as it should be, referring only to data in the period window. I think some other MA's that are defined recursively are influenced by earlier bars, but not 100% sure.. that is perhaps the subject of the next study ;-) Perhaps I might write up an in-depth piece about the various SMAs and what is really going on. Time permitting!

Anyway, good to discuss all this... which was what I was hoping my post would do... but it got a bit derailed.

I'm still keen to hear from others about their use (or otherwise) of WF. For a technique lauded as the bees knees of strategy performance testing, I think it is really important for users to be aware of the hidden pitfalls!

BTW.. since I started this journey I found this blog post... it's interesting, but seems to have died before the final results were published.

https://www.forexfactory.com/showthread.php?t=472656

Cheers,
T.

[tgn55]

Afterthought... Re the discussion of SMA, valid data etc... surely this is exactly WHY we have BarsRequiredToTrade... to ensure that all indicators used by the strategy have indeed "settled down" and are guaranteed to produce valid/correct results before we begin making trading decisions using them! I think in all the issues above we lost sight of this simple concept...

Oh.. and if it isn't obvious, of course I set BRTT within the strategy State.Configure or State.DataLoaded to the max MA period. Basic stuff, really

Cheers,
T.

[Mark17]

I was searching for critiques to WFO and I found this thread. I wanted to add my thoughts.

I've tested a few hundred strategies over the last few months and found it very difficult to pass WF with my desired performance statistics. One thought that repeatedly occurs to me is "why does this matter?" What we're really testing with WF is whether variables that work for one period will subsequently work for the next. On one hand, this should obviously be the case: backtesting looks for variables that work over the backtested period with the hope of trading live and achieving similar success. On another hand, though, how many times do we hear about a mutual fund or asset class that performs well in one year then doing horribly the next? Mean reversion abounds. Should we therefore reject a strategy just because it's not trainable in this way when mean-reversion with regard to performance is so common? Maybe that's too harsh. Maybe we should stick with one set of variables and keep trading it if they produce our desired performance characteristics.

Here's one other critique. WFO chooses the combination of variables that scores highest on our objective function. It does not explore the surrounding parameter space, however. What if 5 scored best but 4 and 6 scored horribly? That would be different than 5 scoring best while 4 and 6 scored pretty well. I'd go for the latter even if the latter 5 scored lower than the former 5. WFO doesn't explore this. Should it?

It doesn't surprise me to hear performance vary so significantly with a slight change in start or end time even if the variables are fixed. Maybe, tgn55, this is a good stress test. In addition to a satisfactory WF equity curve, maybe we should also look at WF equity curves over shifted time intervals. One thing I have seen some software do is look at WFO performance using different ratios of IS:OOS. Perhaps this accomplishes a similar stress function.

[NinjaTrader_ChelseaB]

Hello Mark17,

Welcome to the NinjaTrader forums!

Where you are mentioning exploring the surrounding parameter space, the results do show other parameter sets with their performance score and you can choose the number of best results to keep.

Are you wanting to see results that did not perform well?

[Mark17]

Hello Mark17,

Welcome to the NinjaTrader forums!

Where you are mention exploring the surrounding parameter space, the results do show other parameter sets with their performance score and you can choose the number of best results to keep.

Are you wanting to see results that did not perform well?

What I'm suggesting is that the optimization take into account the consistency of performance in the area centered by that particular variable combination. Performing well in isolation would not score as high as performing well and being surrounded by nearby variable combinations that also perform well.

Please note that I'm speaking of WF in general--not NT specifically.

[tgn55]

Hey Mark17, a couple comments you might find worth pondering.. and exploring.

In my optimisation tests, I find it REALLY useful to use Ninja's Optimisation Graph feature... which enables you to see the spread of results for each parameter setting... not just the customary 10 reported in the top pane. If you have multiple parameters, you can select which ones to view. So... that's cool

Here's what I have learned to do...

Instead of targeting the Max Profit or whatever... I find it is often better to focus on parameter settings that exhibit the LEAST variance in Profit... (with caveat's of course)
Also.. consider this. The process of optimising parameters is very seductive..But, stop and ponder... if you select the MAX of <whatever> ... you are by definition identifying an "outlier" of the range of outcomes. By definition... this is not a "typical" result ... but an extreme. This is why I often resort to a search for a) limited variability, and b) consistency... rather than getting all excited about the Max value. To this end, Ninja's optimisation parameter "Max Strength" is worth a look... it more or less finds the parameter set that produces the "least lumpy" PnL curve, or something along those lines.

Finally, Ninja provides a neat 3D view of the surface the Profit curve for multi-dimensional parameter optimisations... which can give some insight into how things change as you move away from the parameter value that generated the max profit (or whatever)... Cool... just beware that as it maps out a 2D slice of the (possibly) many parameters, WHat it does, which is (was) not documented, is the value displayed is the AVERAGE of all the profit results for all the other parameter settings. Hope that makes sense... I'll try again to explain.

Say you have a strategy with 6 parameters. The 3D graph enables you to pick any two. So - 2 are fixed, and we have 4 parameters each of which will generate various outcomes. To display the 3D surface view, Ninja has come up with just ONE value for the chosen pair of parameters. Which value? There are likely dozens of results that all share the specified pair. Well, it averages all the results that match the selected pair at each point of the 2D plane, and that's what gets plotted.

Hope that all makes better sense.

Bottom line: Ninja has some great tools to analyse stuff. Yes, beware of placing too much confidence in WFO... treat it like "The Pirate's Code"... more like guidelines ;-)

Good luck,
T.