Interpreting Walkforward results

[ShruggedAtlas]

I am having difficulty interpreting my walkforward results. I have not been able to find anything online that talks in depth about what defines good or bad walkforward results.

does anyone have link to in-depth discussion about this topic?

[NinjaTrader_DaveI]

Hello,

While it is out of our scope to discuss what would constitute good or bad walk-forward results, or to point to a discussion of the same nature, I'll be happy to explain any part of the report that is confusing or which seems to be inaccurate. In the meantime, this thread will remain open for other users to comment on what they feel to be good or bad results.

[ShruggedAtlas]

Does anyone know of previous discussions about walkforward results....it seems a bit of an art form.
There is so little discussion of results analysis - plenty on the procedure itself. This leads me to believe that by and large, traders are not using walkforward all that much. why is that?

[koganam]

Does anyone know of previous discussions about walkforward results....it seems a bit of an art form.
There is so little discussion of results analysis - plenty on the procedure itself. This leads me to believe that by and large, traders are not using walkforward all that much. why is that?

I use WalkForward in my tests. What is the issue here?

[ShruggedAtlas]

I use WalkForward in my tests. What is the issue here?

I am looking for ninjatrader specific discussions about interpreting results of walkforward optimizing. There is no issue other than I'm wondering how best to interpret my results. What do I need to be looking for in the results. What results should I consider subpar and what results should I consider encouraging. How is this done? is there a discussion online anywhere about this?

[koganam]

I am looking for ninjatrader specific discussions about interpreting results of walkforward optimizing. There is no issue other than I'm wondering how best to interpret my results. What do I need to be looking for in the results. What results should I consider subpar and what results should I consider encouraging. How is this done? is there a discussion online anywhere about this?

Have you started from the Wikipedia article, which pretty much says what I would have said, even if a bit more elaborately?

ref: http://en.wikipedia.org/wiki/Walk_forward_optimization

[ShruggedAtlas]

yes I have looked at the wiki article and as with most other articles about WFO it simply states what it is, i.e. how it works, and what it is for. Any in-depth discussion about interpretation is not included.

I'm looking to answer questions like:

1. which is better, longer optimization periods or shorter optimization periods and how can you tell from the results?
2. which is better, tight strategy parameters or wide strategy parameters and how can you tell from the results?
3. should one use7+ years of data or is it sufficient to use just a couple of years and how can you tell from the results?
4. which "optimize on..." parameters yield the most robust results and which ones are to be avoided and how could you tell from the results?
5.what does it mean when your results included widely varying parameter adjustments?
6. what does it mean when your results show roughly even mixed wins and losses results from week to week such as: -232, -263-,-44, -146, +100, +148, +198, +20, 0, +228, 0?
7. if optimizing for max. profit factor why does Percent Profitable go down while the number of trades taken also is reduced? (general optimizing question)
8. when I look at the instruments tab on the instrument I did the WFO on, and open the performance window and see the settings tab I am presented with parameters list that do not match my original parameters or even the range of parameters I set the optimization on. What does this mean?
9. Are the optimizer results useful in anyway or should I be looking mostly at the walkforward results?
10. optimized results are much better (as to be expected) than the walkforward results. does this mean curve fitting? does this mean I should optimize more often? won't I be more likely to curve fit by optimizing more often?
11. which strategy parameters should I use once I'm done with WFO. how can I know since they may be different at each of the WFO testing periods.
12. Should I take the last one and then do a backtest using those final parameters on the entire data set of several years?

I could go on and on...and there are no online resources that answer these questions from what I can tell, and most certainly not ninjatrader specific. the wiki and other online sources do not speak in plain language that breaks it down for someone new to this.

I have a simple, yet multi-timeframe and multi-parametered strategy which yields a surprising number of possible variations and nuances that I have to work with. At this rate it will take many months of experimentation to gain an understanding. Surely there must be some resource out there which I can use/buy that can get me there more quickly?

[ShruggedAtlas]

ahhh the silence.

I am beginning my long arduous journey to figure out how to interpret the walkforward feature. In several weeks i hope to have some basic understanding and I will be sure to post my observations.

[ShruggedAtlas]

My approach at this point is to

1. Backtest with my original parameters of which I have, for now, 7 which can vary widely. I'm backtesting on the longest data sets i can get my hands on...that is data from 1/1/2000 to today. (Each backtest on genetic takes upwards of 2 hours.)

on my multitimeframe strategy the parameters include:
a. Long Term timeframe
b. Trading timeframe
c. Oscilator low
d. Oscilator High
e. Entry/Exit oscilator trigger value
f. Protective stop value
g. Oscilator period value

2. Optimize on a range of values for each parameter independently so I can somehow get an understanding of how the parameters effect results for each of the key "optimize on..." catagories such as max % profitable, max net profit, max win/loss ratio and max profit factor. (Each backtest from 1/1/2000 takes up to 10 minutes depending on the range of values in each parameter.)

3. Based on the results for each, I'll pick the parameters that seem to generally give a solid balance of "good" results (whatever that means) between the profit factor, % profitable and win/loss ratio.

4. Once i have a tighter range of values and a better understanding of how the values impact the strategies performance for each "optimize on..." category, I will begin the walk forward process using tighter ranges of parameter values. I will be performing the WFO using all the parameters at the same time with tighter value ranges so the testing should take roughly 2 - 5 hours on each instrument on 15 years data.

5. I will start with longer WFO periods and begin to shorten them.
Example: Optimize on 2 years data and test on one year. Then optimize on 6 months data and test on 3 months. Then optimize on 1 month and test on 2 weeks down to optimizing on 2 weeks and testing on 1 week. I will be looking for some indication that the optimization period length and/or optimization/testing ratio has any bearing on the results.

6. At this point I should have a general sense of which values perform "generally well" in the few instruments I trade. I will at this point observe sim trading in real time and begin incorporating real time ATM management into the code. I will then at that point not be able to backtest the code since backtesting cannot utilize real time ATM procedures.

Does anyone have any thoughts as to if my approach is sound or can be made more efficient (timewise)?

[koganam]

That is quite a number of questions that you ask. I have given some answers in the text of your original post. In reality, each punter must come to their own conclusions as to how well their system works in relation to how they are inclined to trade.

yes I have looked at the wiki article and as with most other articles about WFO it simply states what it is, i.e. how it works, and what it is for. Any in-depth discussion about interpretation is not included.

I'm looking to answer questions like:

1. which is better, longer optimization periods or shorter optimization periods and how can you tell from the results? - Neither. What is best depends more on the variations that exist in the dataset. One can only experiment to find the best parameters here. Periods that are too long can have the effect of masking variations, and so being of not much use. Too short periods may have the effect of catching unstable transient effects.
2. which is better, tight strategy parameters or wide strategy parameters and how can you tell from the results? - Another "it depends on what you mean by tight". Many wide parameters may mean that one is just testing over so big a field that finding good parameters is just a matter of there being so many parameters that some are very likely to be good. On the other hand if one ends up with a large set of good parameters, it means that the system is not too sensitive to inputs. If this is coupled with good returns, that is usually a stable winning system. If one ends up with a very narrow set of good parameters (especially just one or two), one needs to test just outside those parameters. A large drop off of returns just outside such narrow parameters would imply a highly overfit curve. Unfortunately, that is not the whole story. If a very narrow set of parameters provide good WFO returns, it can also mean that you have a very stable system, that need little reoptimization, as long as the market conditions do not change. This speaks to the stability of the underlying test data, rather than to the robustness of the system. If the underlying data changes, the system may collapse. In other words, you may just have run into a very long period of consistent market conditions, that are very well suited to the system.
3. should one use7+ years of data or is it sufficient to use just a couple of years and how can you tell from the results? - You will have to experiment. Long data sets mean that you are more likely to capture a variety of underlying conditions in the data set, but after a while, one reaches a point of diminishing returns in variations.
4. which "optimize on..." parameters yield the most robust results and which ones are to be avoided and how could you tell from the results? - Now, if we already knew that, we would not need to test would we? Generally I make my first test looking for "Max. profit" (not the best criterion), then test that resultant field of optimized set for a more stable criterion, like "Profit Factor", "Expectancy" or "SQN".
5.what does it mean when your results included widely varying parameter adjustments? - If the returns are good, it means that you have an unstable system, that shows good short periods of forward predictability. It means that you have to reoptimize very frequently, and usually also means that the system is so unstable that it will likely eventually completely break down.
6. what does it mean when your results show roughly even mixed wins and losses results from week to week such as: -232, -263-,-44, -146, +100, +148, +198, +20, 0, +228, 0? - You will have to measure the stability of those returns, but it often just means that the system just does not follow through with consistency over your test periods.
7. if optimizing for max. profit factor why does Percent Profitable go down while the number of trades taken also is reduced? (general optimizing question) - That may not be generally true. You may be running into a situation where you are locked around a local optimum that simply just does not have a good dispersion of results.
8. when I look at the instruments tab on the instrument I did the WFO on, and open the performance window and see the settings tab I am presented with parameters list that do not match my original parameters or even the range of parameters I set the optimization on. What does this mean? - That is not my experience, so I cannot comment.
9. Are the optimizer results useful in anyway or should I be looking mostly at the walkforward results? - Optimizer results can provide a good smaller testing set to use for WFO.
10. optimized results are much better (as to be expected) than the walkforward results. does this mean curve fitting? does this mean I should optimize more often? won't I be more likely to curve fit by optimizing more often? - Yes and no. All optimization is curve-fitting to a degree. WFO exposes just how severe this curve-fitting may be. It may pay to optimize more often, if one can find a suitable training period length, but one still needs to be more careful in general: results such as you describe usually imply an unstable system, with parameters that are too wide to really be meaningful. Pretty much, this is the same question and answer as Question 5.
11. which strategy parameters should I use once I'm done with WFO. how can I know since they may be different at each of the WFO testing periods. - You will use the parameters that are spit out on your optimization schedule. The purpose of WFO is to find the training period/execution period combo for an optimization schedule. You use that schedule to optimize, and you use the optimization parameters that you get for the next forward trading execution period.
12. Should I take the last one and then do a backtest using those final parameters on the entire data set of several years? - That is pretty pointless, unless you have such a stable set of parameters, that it already points to the lack of need to reoptimize in the first place.

I could go on and on...and there are no online resources that answer these questions from what I can tell, and most certainly not ninjatrader specific. the wiki and other online sources do not speak in plain language that breaks it down for someone new to this.

WFO is not NinjaTrader specific. NinjaTrader just provides a, better than most, facility for some amount of more easily conducted WFO. Even then, NT does not allow us to optimize the WFO period parameters: we still have to do that manually. We only get to find the optimal parameters for the periods that we have to manually select.

I have a simple, yet multi-timeframe and multi-parametered strategy which yields a surprising number of possible variations and nuances that I have to work with. At this rate it will take many months of experimentation to gain an understanding. Surely there must be some resource out there which I can use/buy that can get me there more quickly?

I have not found any. Most of my responses have been from my own experimentation, and preliminary system analysis to determine a method to go forward.

I know that my answers are a bit imprecise. Unfortunately, that is the nature of the beast. The value and validity of optimization in general, and WFO in particular, is pretty much dependent on what the interpreter wants to do. Granted that bad optimization returns are bad, and show a clear need to rethink the system, most WFO optimization results are to be interpreted by the user. There really are not many fast and hard rules if the WFO is not an abyssmal failure.

[koganam]

My approach at this point is to

1. Backtest with my original parameters of which I have, for now, 7 which can vary widely. I'm backtesting on the longest data sets i can get my hands on...that is data from 1/1/2000 to today. (Each backtest on genetic takes upwards of 2 hours.)

on my multitimeframe strategy the parameters include:
a. Long Term timeframe
b. Trading timeframe
c. Oscilator low
d. Oscilator High
e. Entry/Exit oscilator trigger value
f. Protective stop value
g. Oscilator period value

2. Optimize on a range of values for each parameter independently so I can somehow get an understanding of how the parameters effect results for each of the key "optimize on..." catagories such as max % profitable, max net profit, max win/loss ratio and max profit factor. (Each backtest from 1/1/2000 takes up to 10 minutes depending on the range of values in each parameter.)

3. Based on the results for each, I'll pick the parameters that seem to generally give a solid balance of "good" results (whatever that means) between the profit factor, % profitable and win/loss ratio.

4. Once i have a tighter range of values and a better understanding of how the values impact the strategies performance for each "optimize on..." category, I will begin the walk forward process using tighter ranges of parameter values. I will be performing the WFO using all the parameters at the same time with tighter value ranges so the testing should take roughly 2 - 5 hours on each instrument on 15 years data.

5. I will start with longer WFO periods and begin to shorten them.
Example: Optimize on 2 years data and test on one year. Then optimize on 6 months data and test on 3 months. Then optimize on 1 month and test on 2 weeks down to optimizing on 2 weeks and testing on 1 week. I will be looking for some indication that the optimization period length and/or optimization/testing ratio has any bearing on the results.

6. At this point I should have a general sense of which values perform "generally well" in the few instruments I trade. I will at this point observe sim trading in real time and begin incorporating real time ATM management into the code. I will then at that point not be able to backtest the code since backtesting cannot utilize real time ATM procedures.

Does anyone have any thoughts as to if my approach is sound or can be made more efficient (timewise)?

That is a pretty general approach that is often used.

1. Find the optimized individual parameters.
2. Find out how those parameters behave when the other parameters interact with them
3. Rinse, repeat.

It does not guarantee that we shall find the best parameters, but it often finds a viable set.

I often go the other way to what you want to do: starting with smaller periods and expanding until I reach a viable set. I record those, and keep expanding periods, until I have a set of viable sets. The most viable set often is one where the various initial sets coincide. It is a very manual, intensive and time consuming process.

[ShruggedAtlas]

Outstanding reply. I am very grateful for the time you put into responding to my questions. this is precisely the kind of insight I was looking for. You have given me plenty to think about. Thank you!