Why you should not trust walk-forward results...

[tgn55]

Hi traders,
I've been giving Strategy Analysis a fairly good bashing over the last 2-3 months, and hopefully several improvements will ship with the next release.

That said... this post is NOT about Ninja... it is about the walk-forward process itself... and I hope worthy of your attention. I would value comment from others.

As we all know, walk-forward testing is touted as the gold standard in strategy testing. I do not deny it has some advantages, ie, reduces reliance on hindsight, but, read on...

I first noticed some worrying results a while back, and just recently saw profit change from around $42,000 down to $7,000... with just one minor change in the walk-forward settings and my, that makes a difference to how I view my strategy's performance ! I needed to understand what the heck caused such a huge variation.

Yesterday, I bit the proverbial bullet and did some basic research/testing of the method itself... not my strategy ... and l confirmed my suspicion, which was that the Start Date of walk-forward tests can have a dramatic influence on the outcome. . At least it does in the class of strategy that I am using... other strategies may do better, but let me share my results.

I went for ultra simple... using the Sample MA Crossover strategy. I tweaked it slightly to enable me to run a WF optimisation ... but using the same parameters in each period.

OK, so what you say. That seems pretty dumb!

My objective was to determine just how much influence the walk-forward process itself had on the final result. My test was over about 13 months, with a 28 day test period. (Optimisation period doesn't really matter, since there is nothing to optimise... both Fast and Slow are fixed.)

So... 28 day test period. I ran 4 walk-forward tests, each starting 7 days later. Same Fast/Slow parameters locked in.

Here's a summary of the results (run on GC 12-19, 60 Minute chart)



What I want to draw your attention to is the HUGE variation in overall profit... changing start date by 1 week, can cause Net Profit to go from around $20K down to ~$10k !! A 14 day change made things go from ~$24K down to ~$10K. That is a massive percentage change!
Now, you may point out that since the period of trades is different, we should not be surprised if the overall profit is different... and of course that is true, and in part explains SOME of the variation... but absolutely nowhere near all of it. Yes I checked the individual trades... it just does NOT cut it.

The point is, changing that start date generates a completely different sequence of trades, in this instance (and also in my strategy). It may be that other types of strategies do NOT suffer this rather fickle property... that is what I'm keen for others to provide input on.

So... the conclusion I've come to regarding walk-forward testing of my strategy, is that the final numbers are possibly due to semi-random factors induced by the walk-forward process itself... and nothing to do with my strategy's intrinsic robustness. Dang !

Sorry it's a long post ... but hopefully worth the time taken to read if you are relying on walk-forward tests to "build confidence in your strategy's performance"

Cheers,
T.

[NinjaTrader_ChelseaB]

Hello T,

It is expected that changing the start and end date, optimization period, or test period would change the data used for the optimization iteration and walk forward iterations with a walk forward optimization and would result in different trades.

Below is a link to the help guide on this.

https://ninjatrader.com/support/helpGuides/nt8/en-us/index.html?walk_forward_optimize_a_strate.htm#UnderstandingWalkForwardResults

Use prints to see exactly what is changing.

https://ninjatrader.com/support/forum/forum/ninjatrader-7/strategy-development-aa/99209-date-conditions-not-working?p=791121#post791121

[tgn55]

Chelsea... if you read my post carefully, you will see there is NO OPTIMISATION in my tests. Trust me - I know how walk forward works (been doing rather a lot lately), and as stated, this test uses FIXED Fast & Slow parameters, by adding in a dummy variable which changes, but Fast and Slow are static..

The whole point of my tests was to isolate things as far as possible to just the things that walk-forward introduces... which of course includes closing out trades at the end of each test period, and starting over in the next period, which in real life you might not do, unless your strategy explicitly has close at end of session.

That is why I constructed my test this way It reports on what changes, when EVERYTHING else remains constant (particularly the SMA Cross parameters). What is left is the artifacts of the walk-forward process itself... and that can cause huge variations. Maybe not always... but enough to make developers wary of believing that a walk-forward test can be relied on.

Hope that helps

PS... Just did another DIFFERENT test, that removes Start Date from the equation.
1) Just ran a plain old SMA Crossover backtest between two dates (I used about a year). Result: $18,470
2) Run a walk-forward over the same date range, with the SAME Fast/Slow parameters Result: $11,270

So this represents the change in outcome purely as a result of chopping up the year into 28 day blocks... NOTHING else changed.
Interesting, don't you think?

In a normal walk-forward optimisation, clearly other factors come into play of course.

[NinjaTrader_ChelseaB]

Hello T,

I am slightly confused. A walk forward optimization is an optimization.

You have mentioned:
'you will see there is NO OPTIMISATION in my tests'

May I confirm you are running a backtest and not a walk forward optimization?

That said, I am only referring to the data used for the tests and I am not referring to the values of inputs.

[tgn55]

Not when you tweak the strategy as I said I did in post #1. I am running the walk-forward process, not a standard backtest. I force the system into running a walk-forward "optimisation" that made no change to the fast/slow parameters. I'll attach my hacked version of the strategy so you can see ... it's nothing more than adding a dummy parameter (which does absolutely nothing) to the Sample SMA code... but I can tell a walk-forward to iterate over it, whilst keeping fast/slow unchanged.. like this:
[ATTACH]n1070262[/ATTACH]

Walk-forward setup...


Kapish?

[NinjaTrader_ChelseaB]

Hello tgn55,

That would still be an optimization using the optimization engine and rules even if the input values are not used and are not changing in the logic.

To confirm, you are running a walk-forward optimization, changing no dates or any other settings and simply running the test again with no changes and the results are different, is this correct?

[tgn55]

No... sorry.

Let me break his out a little.
In my last update yesterday, I added in a "PS", where I referred to removing Start Date form the equation, as I performed a new test that I did not include in the original post. It s even simpler/more basic.

First, I ran a standard backtest (of SampleSMACrossover) over a 12 month period, with one set of fast/slow. No optimisation, clearly. Result: $18,000
Next, I chop the year up into 13 x 28 day periods, and run the same strategy, with exactly the same parameters, using my tricked up SMAPlus thingy. And result: $11,000

It is critical that you "get it" that this process does NOT change Fast/Slow ... they remain static. Look at the pic above! Fast: 6;6;1, Slow 16;16;1 Fast is fixed at 6, Slow is fixed at 16. ONLY my dummy variable changes.

But this was an afterthought...it was not the thrust of my original post, which focused more on the dates (which of course change in each walk-forward test period).
Original post was motivated by seeing huge changes in walk-forward results by simply changing the start date of the run... nothing more. I do NOT expect to see identical result when I do this... but neither do I expect to see $40,000 profit turn into $7,000 by changing a year-long walk forward's start date by two weeks!

I replicated this behaviour, although to a less dramatic degree, using SampleSMA, since I know otherwise you would think it had something to do with my code... it doesn't... it is an artifact of the walk-forward process.

Hope that clears up the confusion...

T.

[NinjaTrader_ChelseaB]

Hello T,

This does not clear up the confusion.

Are you changing the start date, end date, optimization period, or test period?

Changes to any of those will result in wildly different results. Different data, different results.

May I confirm you have used prints (or writing to file) as suggested to see when the data starts and the values used in all conditions to see what is causing the different results?

[koganam]

No... sorry.

Let me break his out a little.
In my last update yesterday, I added in a "PS", where I referred to removing Start Date form the equation, as I performed a new test that I did not include in the original post. It s even simpler/more basic.

First, I ran a standard backtest (of SampleSMACrossover) over a 12 month period, with one set of fast/slow. No optimisation, clearly. Result: $18,000
Next, I chop the year up into 13 x 28 day periods, and run the same strategy, with exactly the same parameters, using my tricked up SMAPlus thingy. And result: $11,000

It is critical that you "get it" that this process does NOT change Fast/Slow ... they remain static. Look at the pic above! Fast: 6;6;1, Slow 16;16;1 Fast is fixed at 6, Slow is fixed at 16. ONLY my dummy variable changes.

But this was an afterthought...it was not the thrust of my original post, which focused more on the dates (which of course change in each walk-forward test period).
Original post was motivated by seeing huge changes in walk-forward results by simply changing the start date of the run... nothing more. I do NOT expect to see identical result when I do this... but neither do I expect to see $40,000 profit turn into $7,000 by changing a year-long walk forward's start date by two weeks!

I replicated this behaviour, although to a less dramatic degree, using SampleSMA, since I know otherwise you would think it had something to do with my code... it doesn't... it is an artifact of the walk-forward process.

Hope that clears up the confusion...

T.

That would be expected using any averaging process. Think about it. The WF discards the previous data and starts afresh on each optimization period. That means that the actual values of the moving averages, especially SMA must also be changing, as the starting point for the calculation of the SMA keeps shifting. So, at the very least, the first values of the SMA in the new period are going to be very different from what they would be if the data from the end of the previous period were involved in the calculation. It means that the crossover time-points and corresponding prices will also be different from what they were in the backtest; or even using a different optimization period, even if using the same parameters for the SMA.

[tgn55]

Agreed (I think !). It also excludes a new bunch of bars prior to BarsRequiredToTrade... for each new test period. That will affect things too, clearly.

So... why the "I think" ... well, it goes to the implementation/definition of SMA.
(Note that we have in my tests BarsRequired is always the same... 20 ... but that is an aside)

If SMA(16) really is the simple average of the last 16 bars ... it should not matter. * See footnote
However, if SMA(16) actually uses data points from MORE than 16 bars ago... well... it's really not so simple, is it! Is it using data more than 20 bars ago??

Anyway, that aside... my point is, (and this goes back to my original post)...
1) If you change the start date, you may get a COMPLETELY different sequence of trades ... and sometimes, that makes a HUGE difference (but not always)
2) even without changing start date (ie, my more recent example), the mere process of chopping up a period into discrete blocks can in itself create HUGE changes (but not always), due to the WF slice and dice process itself...

In summary : my post was waving a red warning flag. Given the above, strategy developers need to realise that a walk-forward test is NOT the God of backtesting reliability that many traders and educators make it out to be. I think my $40,000 turning to $7,000 example is surely sufficient evidence! I'm not saying it is completely useless... you just need to not blindly accept the first result you get as gospel. It is more complicated than that. CAVEAT EMPTOR !!

* In my earlier years, I used to lecture at University final year level on this exact topic... fairly advanced math, in which SMA was the most basic kindy kid stuff. So, yeah... I kinda know what I'm talking about

PS Thanks koganam for your input... it was getting a bit of a two-party dialog!

PPS The key takeaway remains.. WF tests introduce a bunch of stuff that can distort your results HUGELY. So.. watch out!

[tgn55]

Chelsea...in my simple, most recent test I am not changing ANY of the things you mention above. That is the one using the SampleMAPlus code.

In my original post, I ran a different test ... and yes, in there I ran a series of 4 normal WF runs, each with a different start date. All this should be clear if you look at the images I have included in my post. Oh... I just scrolled up, and see the pictures I attached in the op have disappeared. That can't help...

Two different tests... not the same. I still see the image in post #5 ... from which you can see I am running a walk-forward. I did not include an image of the backtest... didn't think that would be required.

To try to get you over this confusion, maybe I'll record a video and show you exactly what I did. But possibly not tonight... it's already 2:45 AM where I am...

BTW... what happened to my picture attachments in post #1 ?? Maybe I'll edit and try to reinsert them ...

[koganam]

...
If SMA(16) really is the simple average of the last 16 bars ... it should not matter. * See footnote
However, if SMA(16) actually uses data points from MORE than 16 bars ago... well... it's really not so simple, is it! Is it using data more than 20 bars ago??

Yes, but not if the calculation restarts at 0 every time you start a new period.

[tgn55]

I might do some analysis on this, as it has puzzled me for a while now, just never had the time to dig into it.

From a theoretical/mathematical view, ONLY data points in the moving average window should matter. That said, I'm not sure that is how it is commonly implemented on trading platforms...and that's what I'll follow up... later. It's now heading for 4 AM... think I'm going to bed.

Tomorrow I'll record a video. Not sure why this has caused confusion, but it evidently has.

I am still interested to hear if other traders have hit this phenomenon with other strategies... but unless you do a bit of playing, you might not realise it. In my case, PnL dropping about 80% was a pretty loud and clear wakeup call

[NinjaTrader_ChelseaB]

Hello tgn55,

My recommendation is that instead of a video, you use prints (or write to a file). Numbers always help.

Here is an example post that demonstrates.

https://ninjatrader.com/support/forum/forum/ninjatrader-8/strategy-development/100192-comparing-real-time-historical-and-replay-performance#post100192

[tgn55]

Hey koganam... I ran a test to check if the SMA values get messed up by the WF process... it is summarised in the attached doc.

I cut a little strategy that just dumped SMA values on each bar. Ran it in backtest mode, so I have a baseline reference.
Then, run it as a walk-forward... so the period is chopped up a few times, so I can test the concern you raised re SMA being reset for each new test period.

Bottom line is ... all is well. Now, I think the caveat is this... all is well provided the BRTT exceeds the period of the SMA.

So in my case, I had BRTT=14, SMA period 10. Each test period will therefore ignore the first 14 bars, then calculate SMA(10) with all valid data.
There is NO reference to bars prior to the start of the SMA period.

And... the concern you raised in post#9 is not an issue... all good.

SMA Test.zip

Cheers,
T.