Different walkforward results

[kucharek]

Hi,

I did the three quick walkforward tests. I change nothing but optimization period - 28, 14, 7.
With no parameters change results shall be the same in all three cases but are different.

I use Kinetick basic service.

Please find attachments for details.

Please explain where is the problem.

[NinjaTrader_Jason]

Hello kucharek,

This can be as expected. If you use a different optimization period, you can get different results.

If the optimization period is less but the amount of historical data is the same, the walk forward period is longer. More information regarding walk forward tests can be found at the link below.

http://www.ninjatrader.com/support/helpGuides/nt7/walk_forward_optimize_a_strate.htm

[kucharek]

Hi Jason,

Based on your feedback I did two more walkforward tests. This time I arrange the same walk forward period. Please find attached screenshots of the results.

WF period is the same but results are different.

[NinjaTrader_Jason]

Hello kucharek,

You are using a different setting for 'Optimization period' with different dates. The differences between the tests are not that big. If you like to find out why there is a difference, please analyze the trades in the Trades-tab of the Strategy Analyzer.

If you like to have identical results, please use the exact same settings.

[kucharek]

Hi Jason,

I did as you suggested and still I do not know what and why is happening here.

Please find attached some screenshots.

1. sma1 and sma2
It is your SMA crossover sample strategy. In both cases walkforward period is: 01.01.2009 - 01.01.2012. Why there is extra trade on 16.01.2009? It is not logical for me.

2. bt1, bt2chart1 and bt2chart2
It is my strategy. In both cases walkforward period is: 01.01.2005 - 01.01.2012. Why there is:
- 11.01.2005 and 26.07.2005 trade on the "right" test only?
- 25.03.2005 on the "left" test only?
It is not logical for me.

Please support and explain.

[NinjaTrader_Jason]

1. The first order occurs at different times when I check your screenshot. I see the From date is different in the settings. This is causing the difference in results.

2. Different From dates are used as well. You only get identical results when using the exact same settings.

[kucharek]

Jason,

I am unable to agree with you.

It is different because I change optimization period but walkforward period is the same.

1.
Left test
Optimization period: 30.12.2008 - 31.12.2008 (1 day)
Walkforward period: 01.01.2009 - 01.01.2012
Right test
Optimization period: 31.12.2008 - 31.12.2008 (2 days)
Walkforward period: 01.01.2009 - 01.01.2012

Because I chose no parameters to optimize it shall be no difference how long is the optimization period. Most important is that walkforward period is the same. Strategy runs and open positions in exactly the same period.

2. Situation is identical as described below.

It is simply not logical.

[NinjaTrader_Jason]

The different optimization period cause orders to occur at different times. The walk forward with 2 days optimization period has its first order on 1/15/2009. The other one has the first order at 1/16/2009.

[kucharek]

Jason,

1. sma1 and sma2

1.1.There is only one optimization test at the beginning and one walkforward test. In these cases it is simple backtest from 01.01.2009 to 01.01.2012. There is no optimization periods during walkforward period.
Also, please check on the AUDUSD chart - there is SMA crossover neither on 1/15/2009 nor 1/16/2009.

1.2 There is only one signal to open a position - SMA crossover. It shall have nothing to do with optimization period. Optimization period can not "move" signals.

2.
What about:
- 11.01.2005 and 26.07.2005 trades on the "right" test?
- 25.03.2005 on the "left" test?

[NinjaTrader_Jason]

1. Yes, I understand no real optimization is done since the same parameters are used.

Unfortunately, you do not see the cross of the SMA, because indicators are not plotted on the first 20 bars.

If you compare the Trades-tab, you will see that all trades are the same except for the first one. This is causing the discrepancy.

2. Different optimization periods and dates are used. Please analyze your strategy and the trades to determine why there is such a difference.

[kucharek]

Jason,

1. But I see SMA cross on backtest or the chart.
By the way, why these positions are opened (1/15/2009 and 1/16/2009) if min. bars required is set to 20. If I change it to 60 or 300 nothing changes. Please check it on your site. Maybe there is some bug on my site only.

2. I definitely disagree. Please note - this is a backtest done in walk forward optimization window, nothing more. Test period is exactly the same. Optimization period or strategy itself has nothing to do with it. This is the same strategy, the same period, the same data and the same computer. Please review it in details.

Please explain.

[NinjaTrader_Jason]

1. Unfortunately I am not sure how to interpret your first sentence.

This setting applies to the optimization period.

2. Can you please clarify why are you performing a backtest when you do a walk forward test? What happens if you perform two backtests with the same From and To dates?

[kucharek]

1. When I check SMA 10/25 cross on the instrument chart or in the backtest I can see that there is no trade signal on 1/15/2009 and 1/16/2009. Right signal is on 1/19/2009 only. I am wondering why backtest in walkforward acts differently.

2. I think that we do not understand each other. Is it possible to talk on the phone?
I was working on NT for a while. I was performing backtest and optimizations and everything worked fine. Now I decided to take a closer look at walkforward. Walkforward is nothing more then optimization and backtest (2 in 1) so I decided to do a simple test of it first. I am presenting a case here which is my test if walkforward works ok.

Please note that my case is a walkforward. Atypical one because I have short optimization at the beginning (1 or 2 days) and after that I have one long test (3000 days). But it is still walkforward. In my opinion it doesn't work ok here so I am not sure if it will work ok in more complicated cases.

Please consider that.

[NinjaTrader_Jason]

I cannot reproduce the order that occurs on 1/15 or 1/16 on my end when I perform the same walk forward test, see screenshot.

What NinjaTrader version do you have installed? You can check this at Help-->About. Please make sure you have our latest version installed (7.0.1000.9). You can download and install the latest version from the link below on top of your current version. Make sure NinjaTrader is closed when you install the new version.

http://www.ninjatrader.com/download-registration.php

Are you always using a Daily interval when you perform the walk forward test?

[kucharek]

Jason,

Thanks for checking that. This is a good news.
I have .7 version. I am now installing the new one.

Recently I was using only Daily interval through Kinetick End of Day. Before I had been using MB Trading data through their demo account.

Is it possible that I can have some conflict between MBT data and Kinetick data now? Maybe I shall delete everything and re-install NT from the beginning. What is your opinion?