Walk forward optimization

[bertochi]

I would like to submit a request to modify the way the "walk forward optimization" is being performed at the moment. Josh said (in an old post from 2008) quite rightly that "When you have a walk-forward of 1 week periods it means you end each test at the end of that 1 week. All positions close out at the end of the week." and there is actually no need for that, no need to close the position "at the end of each test period". Say for example, you are running a 1 year walk forward optimization with parameters (optimize 30 days, test 10 days), we (I mean, "I") would like to see the walk forward for the whole year, but if the positions are being closed at the end of each test period, 10 days in this case, and then waiting for a trigger to open a position on the next test period, it defeats the whole idea of the walk forward optimization, at least in my opinion. Anybody has any thoughts on this? Thanks

[NinjaTrader_Jim]

Hello bertochi,

Thanks for your post.

I'd maybe like to flush this idea out a little further before we submit a feature request.

Walk Forward Optimizations are intended to come up with parameters from in sample data to be used to test out of sample data. After calculating another set of in sample data, the trades calculated will be differently. To my understanding, the criteria to close a position would then be different than the criteria that opened it if the positions are not to be closed.

Could you achieve your intended result by setting the Test Period to a long period of time, or is your expectation to run a backtest that changes parameters over time while maintaining opened positions?

For thread reference information on Walk Forward Optimizations can be found here - https://ninjatrader.com/support/help...e_a_strate.htm

I look forward to being of further assistance.

[bertochi]

Hi there,

The positions are always closed at the end of the test period, so in my example, position are closed at the end of a test period of 10 days, and the new test period starts with flat positions waiting for your trigger to enter a new position and again any positions closed at the end of this test period. I don't know if its me, but I do not see how could this be any useful if you want to have an idea how a walk forward optimization over a period of 1 year could have performed. What we need, is at least the option, to leave the positions opened and continue the new test period with the new optimized parameters, as this is how it should be in real time. Makes sense?

[NinjaTrader_Jim]

Thanks bertochi,

I have submitted a feature request on your behalf and will edit this post when I receive a ticket ID.

EDIT: Feature request ticket ID is SFT-3988.

We collect interest in feature requests before determining if the feature should be implemented. For that reason we cannot offer an ETA or promise of fulfillment. Upon implementation, the number for the ticket ID can be publicly found in the Release Notes page of the help guide. I will provide a link below.

Release Notes - https://ninjatrader.com/support/help...ease_notes.htm

[bertochi]

Thank you Jim. I hope this feature gets prioritized and implemented soon, as I do not think it's correct the way the walk forward optimization is being performed at the moment. It is even adding the results for the whole period as a total, but how could you come to any conclusions if your positions were closed at the end of each of the chunk test periods? right? Thank you once again.

[bertochi]

*************** Dear Jim, as regards to your comment: "I am still scratching my head on the usability of an analysis where positions are opened with one set of parameters but closed with another". I can only say that is why there are dynamic indicators such volatility adjusted moving averages or others similar to these, that change their parameters based on market volatility, or trailing stop losses that change their value based on market action. Parameters do not have to be fixed. You do not want to exit the positions at he end of the test periods, you simply want to carry them but with different exit parameters, on the walk forward optimization. Once this is done, the totals shown as a summary will have more meaning. ***************

[koganam]

*************** Dear Jim, as regards to your comment: "I am still scratching my head on the usability of an analysis where positions are opened with one set of parameters but closed with another". I can only say that is why there are dynamic indicators such volatility adjusted moving averages or others similar to these, that change their parameters based on market volatility, or trailing stop losses that change their value based on market action. Parameters do not have to be fixed. You do not want to exit the positions at he end of the test periods, you simply want to carry them but with different exit parameters, on the walk forward optimization. Once this is done, the totals shown as a summary will have more meaning. ***************

What you are describing is a backtest with dynamic exit parameters.

If the test period has no exits, it is not possible to determine the profitability of the test period, so what criteria can be used for walking forward. Walk-forward testing is limited, agreed, precisely because trades must be completed within the test period in order to determine the optimum parameters. So walk-forward testing always uses fixed parameters whose values are optimized depending on what happens in the test period, and then the current supposedly optimal parameters are used n the next immediate out-of-sample period.

[koganam]

*************** Dear Jim, as regards to your comment: "I am still scratching my head on the usability of an analysis where positions are opened with one set of parameters but closed with another". I can only say that is why there are dynamic indicators such volatility adjusted moving averages or others similar to these, that change their parameters based on market volatility, or trailing stop losses that change their value based on market action. Parameters do not have to be fixed. You do not want to exit the positions at he end of the test periods, you simply want to carry them but with different exit parameters, on the walk forward optimization. Once this is done, the totals shown as a summary will have more meaning. ***************

What you are describing is a backtest with dynamic exit parameters.

If the test period has no exits, it is not possible to determine the profitability of the test period, so what criteria can be used for walking forward. Walk-forward testing is limited, agreed, precisely because trades must be completed within the test period in order to determine the optimum parameters. So walk-forward testing always uses fixed parameters whose values are optimized depending on what happens in the test period, and then the current supposedly optimal parameters are used in the next immediate out-of-sample period.

[bertochi]

Dear Konagam,

I was replying to Jim's question about "the usability of an analysis where positions are opened with one set of parameters but closed with another", with an analogy, and mentioned dynamic indicators where their parameters change depending on market action.

If the walk forward optimization "as it is right now" is helpful to you, then that is good. I am requesting the option to not exit positions at the end of the test period to make the analysis more meaningful at the end of your optimization, and to be able to compare such optimization to a normal backtest. If this feature is added as an additional option, then no harm is done to anyone, and potentially could help many.

KInd regards

[bertochi]

Konagan "If the test period has no exits, it is not possible to determine the profitability of the test period"...

Are you sure you are not confusing the "optimization period" with the "test period"? The optimization period must have exits to determine the best parameters, but the test period can be carried out without closing the positions at the end of such test periods, and then continue the new test period with your open positions and a new set of optimized parameters.

Regards

[koganam]

Konagan "If the test period has no exits, it is not possible to determine the profitability of the test period"...

Are you sure you are not confusing the "optimization period" with the "test period"? The optimization period must have exits to determine the best parameters, but the test period can be carried out without closing the positions at the end of such test periods, and then continue the new test period with your open positions and a new set of optimized parameters.

Regards

Sometimes the nomenclature can get confusing, but yes, if that is the way that you prefer to put it. The in-sample period is used for optimizing (in some descriptions, called the test period), and the out-of-sample period is used for testing (in some descriptions, called validation).

But I agree with you, that what you write is clearer (and also somewhat the more often used nomenclature), so let me restate it in your style. "If the optimization period has no exits, it is not possible to determine the profitability of the optimization period, so that those criteria can be used for walking forward. Walk-forward testing is limited, agreed, precisely because trades must be completed within the optimization period in order to determine the optimum parameters. So walk-forward testing always uses fixed parameters whose values are optimized depending on what happens in the optimization period, and then the current supposedly optimal parameters are used in the next immediate out-of-sample period."

[koganam]

Dear Konagam,

...

If the walk forward optimization "as it is right now" is helpful to you, then that is good. I am requesting the option to not exit positions at the end of the test period to make the analysis more meaningful at the end of your optimization, and to be able to compare such optimization to a normal backtest. If this feature is added as an additional option, then no harm is done to anyone, and potentially could help many.

KInd regards

Maybe I am just confused as to how it would work. How will the optimal parameters be determined and walked forward? Curious, not challenging. I am always up for learning something new.