WalkForward

[neb1998]

Is there a walkforward.cs file like the Genetic Optimizer.cs file available as a reference for those wishing to write a new walkforward engine?

Thanks,

Ben Kane

[NinjaTrader_Austin]

Ben, there is not currently a walkforward.cs file to base a new walk-forward engine off of.

[neb1998]

Thanks Austin. I really wish i could find a way to export to .txt all iteratons from the WF runs, both insample and out of sample. Is this possible with custom code?

I have derived a way to split the run into 2 parts via the actual strategy (IS versus OS) but the fitness function is still basing itself off the total of the run, like just 1 part existed.

Obviously the WF splits the data into IS and OOS but i do not see any commands anywhere to use streamwriter or print statements to export the performance data from the IS iterations and OOS "backtest" for each iteration that occurs during the WF.

[NinjaTrader_Austin]

I will have someone get back to you on Monday.

[NinjaTrader_JoshP]

neb1998,

The way it works is that the code will process the optimization period to figure out values to use on the test period. Then it will process the test period. It will then move forward and use that test period as a new optimization period and then get a new test period after that. Everything is done in a very step-wise fashion. Unfortunately there is no differentiation you can make from NinjaScript to know if it is an optimization period being processed or a test period being processed. What you can do is just write out via StreamWriter and if you've already seen the dates being printed out in that file already then you know it is a test period. If it is the first time you are seeing a date then it is likely the optimization period.

[neb1998]

That sounds promising actually josh, I Think i can get something to work.

That would be a nice function to add in the future. Or just the ability to actually see the iterations rather than the "generations" in the reports viewer. I know it takes a bit more memory, but as an option you can really get a lot more information from the iteration viewer rather than from each generation.

It appears ninja shows the report in the optimizer for the best result of each generation anyway. Based on fitness i assume.

Rarely is a fitness functions top scores what we usually trade, so its nice to have more info rather than less.

Thanks!

BK

[neb1998]

Iterations

Josh, I have my streamwriter set to print the results of each period/run when the last bar is reached. The issue i am having now is the iterations from Insample to out of sample do not match. Meaning the inputs from IS != OS. Which means either ninja is changing the inputs during the test period or the results i am printing are incorrect.

Ninja appears to run all the optimizations on the Insample data and then compare all the results to the out of sample test period instead of going one iteration or generation at a time...This compromises the entire concept in my mind as the fitness function is based only on the insample data and does not try to improve with each iteration / generation.

Any thoughts?

[NinjaTrader_JoshP]

Not sure what you mean exactly. If you set the same number of dates for the optimization period and the test period it will be the same dates. I suggest first running through and just printing Time[0] in OnBarUpdate() to see the output sequence to see how it works.

[neb1998]

Hmm

Josh,

I did what you said about the times, and what i am noticing though is that its not holding the inputs after the first iteration for the actual WF.

I printed this out in the attached File, you can see clearly that my inputs do not remain the same from day 1 to day 2. I am using a optimization period of 1 day and test of 1 day.


So 6/2 should have the same inputs as 6/5 so test the WF...nowhere in the file are the inputs the same for this time period.

[NinjaTrader_JoshP]

neb1998,

Not sure I follow. When you run through an optimization period you are running with a bunch of different parameter combinations. At the end you get one set of parameters which get pushed through to the WF test period. That test period runs with a single parameter set.

It is much easier seen if you just leave optimization and test periods to 28 days and run off of monthly counts. You will see it runs through the optimization period however many times your optimization requires to run through the parameter combinations. Then it will go forward and do one WF with the single optimized parameter set. Then it will go and optimize a bunch of times for that month and push that to the next WF set.

[neb1998]

Ah ok. So each iteration / generation is not individually tested on the the test period? That would make sense from what i am looking at. So basically the optimizer looks for the best settings within an optimize period (over multiple generations) and then applies the best results to the test period - and repeats.

This is indeed a WF, however the fitness function is restricted to the few actual WF runs that are processed. Instead of having every iteration first run through the optimization period, then test period then to next iteration etc.

That would allow the insample/outofsample results to be compared more directly.

[NinjaTrader_JoshP]

The fitness function you choose is how the optimizer decides which is the best parameter set for the WF test period. That test period just uses the best parameter set found in the optimization period as determined by the fitness function or "Optimize on..." parameter. The results are shown as-is for the test period.

A WF is not meant to test every single optimization iteration in the WF period. That would defeat the whole purpose of the WF. A WF is designed to not have any optimization done on the test period and just use whatever parameter set it had before. The reason it is done this way is to eliminate any curve fitting problems that would arise from optimizing on the data you are testing against.

[neb1998]

Sorry if i was misleading, i meant the optimizer runs on every iteration of the optimized time frame. But also applies the inputs from each iteration against the OOS data. and generates a separate report. Each iteration moving forward then is only ranked by the optimization that takes place on the INsample data. The out of sample window is truly just a backtest, no optimization done. But then the fitness function derives the performance from the Aggregate totals of the Insample and "backtested oos window" which then provides better OOS results (if your strategy has potential) without curve fitting. This is how the grail system works, which tradestation just bought. Its very efficient for WF tests.

[NinjaTrader_JoshP]

Guess I am not quite following how adding in sample to your performance report improves your OOS results.

When you walk forward you have a bunch of in sample periods and their corresponding OOS results. The OOS are what determine your strategy's performance. The in sample iterations do not impact this performance at all since you would think of those iterations as testing iterations and not actual trading runs.

What I am saying is that the in sample iterations already run the fitness function for each in sample period. The fitness function is not applied to the OOS as it would not be applicable there. The OOS is just, like you said, a simple backtest and the end result of your WF is the aggregation of all the OOS periods at the end.

[neb1998]

The improvement comes from the optimizer rating the Aggregate Totals from the Insample and out of sample window while only optimizing on the insample window. Say insample you have 20 trades at 60% profitable and out of sample you have 10 trades at 20% profitable. The combined result will be between the 60% and 20% which will lead to a lower fitness function rank than say a system that trades 20 trades at 60% and 10 at 55%, which are much closer in comparison. Also more than anything what we are looking for in a WF is the actual equity curve and the flatness of it. This can be derived from a number of formulas but the actual results from every parameter set insample applied them to the "backtest (OOS window)" must be provided.

So you have say a NetTotalTrades and then Total Trades, with the total trades containing the total of insample and OOS.

Again, the OOS window is never optimized, but the report of its generated values are used to refine the Insample Optimization to create a PF, sharpe, other metrics etc that better match the insample window to the out of sample window.....