Walk forward analyzer - different overall results from same strategy

[Tomass]

Hello, I have a problem with walking forward analyzer results...

I used your SampleMACrossOver...

Conditions inputs, dates, etc... - everything is the same in both cases. Only thing I have changed is
Test period...
for first case I used Test period = 360
second case I used Test period = 180...

Because there is no inputs to be optimize and both strategies are same and all other conditions are same, I would expect, that in overall, I will get same final result – same net profit and same number of trades...(walk forward periods will be different of course, but because dates, all inputs, variables and strategy conditions are same, they should give both same overall final results at the end?)

(Test period 360) : from 22.8.2015 to 22.8.2018 = 43280 trades
(Test preiod 180): from 22.8.2015 to 22.8.2018 = 43379 trades

net profit is different too of course...Please look at the picture – I highlighted both cases that in same setup and time period it gives different numbers...

What I am doing wrong?

[NinjaTrader_ChrisL]

Hello Tomass,

Thank you for the post.

I ran the same optimization and my trade numbers are different as well, so this is expected. In a walk forward optimization, an optimization is run on the "Optimization period" and a test is run on the "Test period" after that. To see the trades taken on each test period, go to the "Walk forward" tab towards the top.

Please let me know if I can assist further.

[Tomass]

I am sorry, but I still do not understand it...Both proven cases are done on Test period...Both strategies are totally same with no inputs variables (nothing is changing in them), Setups are totally same... But one of them is taking more trades. Why?


case Trade period 360 : 19.9.2015 - 12.9.2016 = 15407 trades
case Trade period 180: 19.9.2015-16.3.2016 = 7721
and 17.3.2016-12.9.2016= 7800

We have different testing periods, but If I have same instrument with same inputs and same variables with same strategy it should give same results, no matter for how many slices we divide test period...if we put it all together again, it should give same number. Because strategy is same, all conditions are same and test period is same:
so case Trade period 360 : 19.9.2015 - 12.9.2016 = 15407 trades
and Trade period 180 = (19.9.2015-16.3.2016 )+(17.3.2016-12.9.2016)= 7721+7800 = 15521....it should be 15407... I am testing same strategy with same parametres on same test period 19.9.2015-12.9.2016, but results are different by 114 trades!!! can you see that there is a problem?

In testing period from19.9.2015 - 12.9.2016 should be same results, because strategy is constant (nothing is changing) and I have a same strategy tested on same period...So why they are different?... Same strategies tested on same condition in same tested period are giving different results, why? And which one of tests is wrong? Because as my photos shows - same strategy is taking signal in one case and same strategy is not taking signal in second case, but strategies and market conditions, signal and time period are same? I am very confused by this...

[NinjaTrader_ChrisL]

Hello Tomass,

Thanks for the reply.

The total number of trades for an optimization are different because of "double counting". If you have a data set of 4 days, your Optimization period in days is 1, and your Test period in days is 1, it will optimize on day 2 and report the number of trades for that day, say it's 1 trade. After that, it will optimize day 2 and test day 3, giving 2 trades for day 3. After that, it optimizes day 3 and tests day 4, giving 5 trades for day 4. That gives a total of 8 trades.

Now do an optimization where your Optimization period in days is 1, and your Test period in days is 2. It will optimize on day 1, and add the trades for days 2 and 3 together, 3 trades. After that, it will optimize on day 2 and test on days 3 and 4, yielding 7 trades. Now we have our total amount of trades for each test period at 10 trades.

The page below is for NinjaTrader 8, but it does go into more detail about the results of a walk forward optimization:

https://ninjatrader.com/support/helpGuides/nt8/en-us/?walk_forward_optimize_a_strate.htm#UnderstandingWalkForwardResults

Please let me know if I can assist further.

[Tomass]

I am sorry, but I still see difference...your scheme (please see picture1) on ninja pages shows, that Test period is same as difference between 1 optimization period start and 2optimization period start...

We have optimization period = 1, Test period = 1 which means that second optimization period start = 1+1

We have optimization period = 1, Test period = 2 which means that second optimization period start = 1+2


and so on....

my words are proved by results of walk forward test ( please see picture2)... Optimization period = 10, Test period = 360
1 optimization period starts 22.8.2015
2 optimzaziton period starts: 22.8.2015 + Test period which is 22.8.2015+360 = 16.8.2016...Exactly how it is in Ninja optimizer results...

From that logic based on your scheme and proven by Optimizer results I made
picture 3 which shows how it should ideally work...

You can see two setups...
Optimalization period = 1 Test Period = 1
Optimalization period = 1 Test Period = 2
they are both giving same number of trades!!!!

because we do not do any optimalization (parametres on your SampleMACrossOver stay constant and everything is same in both cases), we should get same number of trades + same net profit in both cases as described in my posts below......


And it makes sence too...If you have 1 period, with lets say 4 trade opportunities...It will be still 4 opportunities in this period, no matter for how many slices you divide that period . Once you count all these slices back together, you shloud get number 4... ( JUST TO REMEMBER - our strategy is constant, no optimizing and everything stay same...).

So it gets me back to posts before - as you see from this post, number of results and net profit in cases described in below posts, should be same...Why are they different and where is the mistake?

[NinjaTrader_ChrisL]

Hello Tomass,

Thanks for your patience. I did a bit more digging on this subject and I found that the missing trades are coming from a difference in the amount of available data for the SMA indicator. When you have two days of data, for example, the SMA values will be different at the beginning of the next day than a single day chart.

My two screenshots explain this further. Screenshot 1 shows the SMA values going from the 14th to the 15th. Screen shot 2 shows the beginning of the SMA values from the start of the session on the 15th. The Screenshot 2 chart has an extra crossover that the Screenshot 2 chart was not able to calculate.

https://www.screencast.com/t/0kGQ3EeXS - Screenshot 1

https://www.screencast.com/t/G7p9oY08M1Y8 - Screenshot 2

Doing this same thing over bigger periods will cause the behavior to happen exponentially. In conclusion, a day-by-day chart will give you only the possible trades and not account for prior days data, and using multi day charts will produce results that are closer to real life.

Please let me know if you have any questions.