WFO optimization / test period confusion

[ours_solaire]

Hi,

I'm confused by the way the WFO in NT version 8.0.15 works.

I have a system that makes one trade each trading day, so keeping track is easy.
If I set the optimization period to say 10 days, and the test period to 30 days, then, according to the documentation provided by NT here, I would expect to get a nice contiguous series of non-overlapping test periods each 30 days long WITH NO GAPS!!

Alas, this is not what I am seeing when I run the WFO... For a 2 calendar year period, I would expect to see a total of roughly 490 trades (250 per year minus initial optimization period). I am seeing only 323 trades though. When I look at each test individually, I can see that the optimization period is included in the test period, and is therefore excluded from the test, leaving a GAP of 10 calendar days between each test period. This is not supposed to be the case according to your documentation which shows nice adjacent test periods without gaps. (See top part of image).

In addition, when I right click on a test and select View Optimization Results, the result is a series of zeroes (i.e. unpopulated Trade Collection ??) See lower part of image for detail.

Then, just for fun I tried setting the Optimization Period to 30 days and the Test Period to 10 days, and I found that basically NO trades were taken for any of the test periods. This can't possibly be right.

Could someone from NT please explain why the WFO behaves this way? Thanks in advance.

[NinjaTrader_PaulH]

Hello ours_solaire,

Thanks for your post.

It looks like you are using 1440 minute bars as daily bars and you say that your strategy trades once per day. What are your "BarsRequiredToTrade" set to?

[ours_solaire]

Hello Paul,

Yes, I am using 1440M bars as a substitute for Daily bars, but the actual trades are made on a secondary data series set to 5 minute bars so that the system is flat before the market close.

I have tried changing the BarsRequiredToTrade, but with little effect. But it is a small number, around 3.

[NinjaTrader_PaulH]

Hello ours_solaire,

Thanks for your reply.

Can you advise what your entries per direction and entry handling is set to?

We would like to test your strategy to see the behavior on our end, if possible, please send to PlatformSupport[at]NinjaTradaer[dot]Com, mark the e-mail Atten: Paul and include a link to this thread.

[NinjaTrader_PatrickH]

Hello ours_solaire,

Thank you for your patience.

The same behavior can be seen on the SampleMaCrossover.

Can you confirm that trades should of been taken when you test the same periods with the same settings on a Backtest?

I do not see any overlapping or gaps between Test Periods in your screenshot nor in my testing. Can you expand on this item?

I look forward to you response.

[ours_solaire]

Hi Patrick,

I sent my script to Paul a few days ago as requested, but have not heard back from him yet.

Yes, the strategy trades once per day every day the market is open.

The periods the WFO uses are not the issue. The issue is that for a 30 day test period for example, the WFO test should return 22 or so trades fairly constantly. However, as you can see from the screenshot, the number of trades is quite variable, ranging from zero up to 16 or 17. If you drill down into a test period, you find that the system only starts trading AFTER a number of days that seems to correspond with the optimization period. So in the case of a 10 day optimization period, nothing happens for the first 10 days of the test period, i.e. a GAP of 10 days in the test period.

As I mentioned in my previous post, if you make the optimization period longer than the test period as a limit case, you find that basically nothing happens, very few trades are made, which can't be correct, since using a longer optimization period is sometimes where the optimum result can be found.

It would seem to be a relatively quick fix to get the backtester to start at the correct date. Please keep me posted.

[NinjaTrader_PatrickH]

Hello ours_solaire,

Thank you for your response.

We did in fact receive the strategy and tested it. Can you confirm if you take the exact same settings for the optimizations that return no results and backtest them if you still see the same behavior? I want to confirm whether this is solely seen in Walk Forward Optimization or is present in backtesting as well.

I look forward to your response.

[ours_solaire]

Hi Patrick.

I can confirm that the strategy performs as expected for single backtests as well as regular optimizations (i.e. it makes one trade per day every day).

It is only when I use the WFO that I see this irregular trade frequency from the strategy.

[NinjaTrader_PatrickH]

Hello ours_solaire,

Thank you for your response.

Please send me the settings for the Walk Forward Optimization and the settings of the result you tested as a Backtest that did not match.

Your original screenshot does not show the parameters you use for 'Period' and 'ATR Stop Loss' in the Walk Forward Optimization.

I look forward to your response.

[ours_solaire]

Hi Patrick,

The ATR Stop Loss setting has no bearing on the number of trades, and the Period setting only a minor impact for the range I am looking at, which is from 3 to 15. This range was used for the WFO in the screenshot.

I have run the WFO again using Period 5 - 15 as a range, everything else fixed. BarsRequiredToTrade = 3. The optimisation / test period was left at the default setting 10 / 28 days. This gives me a total of 252 trades for the 2 year period I'm looking at.

The fixed Backtest for the same 2yr period, and using a Period setting of 10 gives 503 trades, as expected for a once a day trading system.

See the attached CSV files of the summary screens.

[NinjaTrader_PatrickH]

Hello ours_solaire,

Thank you for your response.

I have run the WFO again using Period 5 - 15 as a range, everything else fixed. BarsRequiredToTrade = 3. The optimisation / test period was left at the default setting 10 / 28 days. This gives me a total of 252 trades for the 2 year period I'm looking at.

The fixed Backtest for the same 2yr period, and using a Period setting of 10 gives 503 trades, as expected for a once a day trading system.

Your strategy is running a 10 day optimization and then a 28 backtest in the Walk Forward Optimization, this is not the same as running a backtest over the entire data from 01/01/2016 to 01/01/2018.

You need to pay attention to the 'Start date' and the 'End date' of the Walk Forward Optimization results to then set that same 'Start date' and 'End date' in the Standard Backtest.

Please let me know once you have made these changes and tested if you still see the same results.

[ours_solaire]

Hi Patrick,

I see what you mean in that each WFO test period is its own standard backtest. The issue is to do with the strategy not having sufficient bars to do anything due to the Period setting being large, thus reducing the number of trades over the WFO Test Period.

This is a problem for anyone trying to use WFO on a strategy that trades often but uses a large value for Period. Perhaps something you can look into eliminating the "dead" period at the start of each WFO test period?

[NinjaTrader_PatrickH]

Hello ours_solaire,

Thank you for your response.

I will forward your request to our development team.

[NinjaTrader_PatrickH]

Hello ours_solaire,

Thank you for your patience.

This feature request has been assigned the following internal tracking id: SFT-3533