How to optimize the Optimization and Test Periods?

[khoga]

Hello,

I've been spending alot of time with the Walk Forward optimizer lately and was wondering how to best to automate figuring out the optimal time periods for the walk forward.

Setting up the parameters in the start;end;step format is great, but having to manually adjust the optimization period through a small number to larger every time, means I need to maintain records of the run each time...is there an easier way to try multiple optimization time periods in the same manner in which we run through the parameters?

If not, then how best to "guess" what the best time periods are for the walk forward? Is there a study or white paper somewhere that shows data beyond a certain time frame is considered irrelevant? (I've read alot of papers along these lines and haven't seen something like that yet...)

Historically when I've optimized, I took slices of the time period (e.g. each month in a year) and ran the backtests on each month, giving me a sample of 12 data points for the performance. Do this over 3 years, you've 36 data points and have a legitimate statistical sample (i.e. avg PF, std dev PF, avg period profit, etc.) This has been a great way to generalize a model to historical data. Currently, though, this requires me running 36 different back tests, though, to get the data. This is where an open API would be very helpful - allow people to create their own version of optimization, which is a very broad academic topic!

Any help is appreciated and happy new year!

[NinjaTrader_Brett]

Hello,

Thanks for the forum post.

I will leave this open for other members to respond on however unfortunately this would not be officially supported.

_Brett

[khoga]

optimizing not able to optimize, then?

I understand that the testing period break outs aren't supported, but the walk forward not being able to optimize on the periods seems like it's a missing link that now needs to be manually optimized, no?

[NinjaTrader_Brett]

Hello,

To make sure we are on the same page.

You are wanting to optimize this specific value:

Test Period(Days)?

I look forward to assisting you further.

[khoga]

Both periods, actually

Brett,

Thank you for taking the time here, I know these types of questions pop up all the time!

Yes, optimizing on the Optimize Period and then also the Test Period.

For example, maybe my model is overfitted past 30 days of optimization, but underfit at 20...how would I know, except through trying repeated values in the Walk forward optimizer? Also, this would keep optimizations potentially shorter - if I only need 30 days, then I don't need to wait for it to run on the last two years.

For the Test Period, if I guess the wrong value here, then I could throw out a perfectly good model, simply because I looked forward to far, or not far enough. Maybe the model is suited to a 5 day look ahead, but not farther? Maybe suited to a 20 day look ahead, but setting it up for 5 days deflates the momentum of letting it run the full 20?

I'm postulating here, as I'm seeing different results for different periods of Opt and Test...some of them good and others not good. Currently, I'm manually honing it down to what the best Opt and Test periods seem to be.

This is only one way to use the walk forward, though. Some people may like creating a model that fits to a specific timeframe, in which case the status quo works great. I, however, don't care what the time frames are and would rather take the better performance and remain period agnostic.

Does that help explain my thought better?

[NinjaTrader_Brett]

Understood, however in this scenario I suspect you are optimizing your optimize run?

As such fitting your value to look best for the test data and then defeating the purpose of the walk forward test?

If you feel that it still needs to be added I will ask development to add it to the suggestion list.

-Brett

[khoga]

Apologies for the delayed response!

Brett,

I'm looking at this from a time series perspective and there are two main concerns when looking at TS from a mathematical/practical point. The first is overparameterization. If we can capture all of the necessary information required for an accurate prediction in 10 days worth of history, then we shouldn't be including data beyond that. In fact, including data beyond that could cause a decrease in accuracy. In other words, there's an optimal amount of data to include before too much data muddies the water.

Second, depending on the model, knowing how far out it accurately forecasts is paramount to running your model. Some models might only really predict the next day while others may capture a longer trend and be valid for several days or weeks. In developing the logic behind the strategies, it's not apparent how far into the future the prediction will hold before the variance makes the prediction invalid. In AR/MA/ARMA models, you get at least a sense of the predictive strength based on the order selection. That's the analogy I'm looking for here.

The fact that the walk-forward runs the same model so many times eliminates the worry of over-fitting, to a degree. Thus, finding the optimal time frames would be a huge advantage, in my humble opinion.

Let me know if I'm being clear or if you've any questions,
Tim

[NinjaTrader_Brett]

Thanks for the feedback.

Happy Trading!

[koganam]

Brett,

I'm looking at this from a time series perspective and there are two main concerns when looking at TS from a mathematical/practical point. The first is overparameterization. If we can capture all of the necessary information required for an accurate prediction in 10 days worth of history, then we shouldn't be including data beyond that. In fact, including data beyond that could cause a decrease in accuracy. In other words, there's an optimal amount of data to include before too much data muddies the water.

Second, depending on the model, knowing how far out it accurately forecasts is paramount to running your model. Some models might only really predict the next day while others may capture a longer trend and be valid for several days or weeks. In developing the logic behind the strategies, it's not apparent how far into the future the prediction will hold before the variance makes the prediction invalid. In AR/MA/ARMA models, you get at least a sense of the predictive strength based on the order selection. That's the analogy I'm looking for here.

The fact that the walk-forward runs the same model so many times eliminates the worry of over-fitting, to a degree. Thus, finding the optimal time frames would be a huge advantage, in my humble opinion.

Let me know if I'm being clear or if you've any questions,
Tim

Might a little Google Fu help?

https://www.google.com/search?q=walk+forward+optimization