WFA Date Range Errors?

[JustCallMeDan]

Edit: Workaround is here: http://www.ninjatrader.com/support/f...47&postcount=8

Strange one this. Using SampleMACrossover, I have set up a walk forward analysis with an optimisation period of 672 days (96 weeks) and a test period of 84 days (12 weeks).

Given I have data ranging from 15/05/2006 to 20/08/2010, and I work out the corresponding date ranges (ignoring the final, incomplete walk forward) to be:

1. Optimisation Window: 15/05/2006-16/03/2008; Walk Forward: 17/03/2008-08/06/2008
2. Optimisation Window: 07/08/2006-08/06/2008; Walk Forward: 09/06/2008-31/08/2008
3. Optimisation Window: 30/10/2006-31/08/2008; Walk Forward: 01/09/2008-23/11/2008
4. Optimisation Window: 22/01/2007-23/11/2008; Walk Forward: 24/11/2008-15/02/2009
5. Optimisation Window: 16/04/2007-15/02/2009; Walk Forward: 16/02/2009-10/05/2009
6. Optimisation Window: 09/07/2007-10/05/2009; Walk Forward: 11/05/2009-02/08/2009
7. Optimisation Window: 01/10/2007-02/08/2009; Walk Forward: 03/08/2009-25/10/2009
8. Optimisation Window: 24/12/2007-25/10/2009; Walk Forward: 26/10/2009-17/01/2010
9. Optimisation Window: 17/03/2008-17/01/2010; Walk Forward: 18/01/2010-11/04/2010
10. Optimisation Window: 09/06/2008-11/04/2010; Walk Forward: 12/04/2010-04/07/2010

Now, NinjaTrader reproduces this, as expected, but things get a little weird when I look closer at the trades for the different periods. It seems as though the simulations don't stick properly to these date ranges. When studying the 'Trades' panel for a specific Optimisation, occasionally a trade taking place entirely prior to the start date appears to be included, and this seems to happen REGULARLY for Walk Forward trade sets!

What's going on!? Why is the strategy operating outside the specified date range?

Note: I am using the Default 24/7 session template as I am studying Forex (old GAIN data that I still have from prior to their departure) - I'm sure people using tighter session templates might easily never come across this.

Edit: Workaround is here: http://www.ninjatrader.com/support/f...47&postcount=8

[JustCallMeDan]

Screenshots

Added screenshots highlighting the issue

[NinjaTrader_RyanM1]

Hi Dan,

Unfortunately this is a known limitation. When using session templates that are 24 hrs with times cross multiple days (e.g. FOREX session template), it is expected that doing things like backtests will include trades from seemingly the date before and the date after. This is because those dates are required to get the full trading session as defined by the session template.



When running walk forwards with the FOREX session template (or similar), each walk forward period may double count trades on "transition" dates. This is a limitation.



Example scenario:

* First period is 1/29 to 2/25. Second period is 2/26 to 3/25.
* First period counts trades from 1/28 to 2/26.
* Second period counts trades from 2/25 to 3/26.
* There is an overlap of two days of 2/25 and 2/26 being double counted between each walk forward period.

[JustCallMeDan]

Hi Ryan,


Thanks for your response. Is there a plan to fix this?

Although I'm sure there is a reasonable explanation as to where this limiting behaviour has come from, from an end user perspective it is basically a failure to 'do what it says on the tin'. To be honest, it's the first time I've come across such a problem in NinjaTrader!

[NinjaTrader_RyanM1]

This is on our list but unfortunately no time frame is available as it would require changing other areas in the platform that work with session templates.

[JustCallMeDan]

That's a real shame.

Perhaps you can clear up my understanding, so that I can work around this limitation - I do not in fact expect to trade 24/7 with any of my Forex strategies, I actually aim to trade only during restricted hours on weekdays.

1) Do session templates (aside from the 'limited' 24/7 ones producing the errors above) effectively limit the data that gets 'fed' into the strategy? As in a weekday 08:00 - 16:30 session template would ensure no DataSeries values get passed to the strategy which do not fit inside this template? I can see the obvious processing benefit of this approach.

2) If so, how does this effect indicators the strategy may use? Does it result in the possibility of, for example, an SMA on 1 minute data of period 5 being calculated from:

Friday 16:27; Friday 16:28; Friday 16:29; Monday 08:00; Monday 08:01?


Given the Forex market is 24/7, this is the kind of behaviour I wish to avoid - although I would only generate trading signals during certain hours, I wish for the indicators within the strategy to essentially operate on 24/7 data.

[NinjaTrader_Bertrand]

Correct, the session templates would limit the data seen by the strategy which would in turn influence indicators using this data for their calcs - for your needs you would likely work on the FX template to get all data and just process signals then in your desired daily time range using a time filter.

[JustCallMeDan]

Fix!

Please note the following fix... well... workaround. Nonetheless it works.

My concern with this was that adding a specific date filter to the strategy variables (essentially needlessly duplicating the Time Frame date ranges from a user perspective) would simply not help with a walk forward where multiple different date ranges will be used automatically, and there is no way to 'roll forward' the new strategy parameter correctly without switching to doing walk forward analysis entirely manually.

So, hunting around with Intellisense, I find the following useful variables under NinjaTrader.StrategyBase:

BackTestMode
BackTestFrom
BackTestTo
TestPeriod
OptimizationPeriod

Logging these values and running a walk forward reveals what specific values they take for different parts of the walk forward. It's all rather straight forward. Oddly enough, BackTestFrom doesn't change when BackTestMode changes from "Optimizer" to "WalkForward", but BackTestTo does - so there needs to be a little code to correctly calculate the actual walk forward start, but essentially adding this tiny little code snippet to your strategy in OnBarUpdate() right at the start, will prevent this error completely:

[FONT="Courier New"][SIZE="2"]
if ((Time[0].Date > BackTestTo) || (Time[0].Date < (BackTestMode.ToString() == "WalkForward" ? BackTestTo.AddDays(1-TestPeriod) : BackTestFrom)))
	return;
[/SIZE][/FONT]

You can check this by adding the below and then commenting out the above - errors are then regularly thrown in the log.

[FONT="Courier New"][SIZE="2"]
protected override void OnTermination()
{
	Log(ToString() + "; From " + (BackTestMode.ToString() == "WalkForward" ? BackTestTo.AddDays(1-TestPeriod) : BackTestFrom).ToString("dd/MM/yyyy") + "; To " + BackTestTo.ToString("dd/MM/yyyy") + "; Mode " + BackTestMode + "; Test Period = " + TestPeriod + "; Optimization Period = " + OptimizationPeriod, LogLevel.Information);
	Log(ToString() + "; Trades = " + Performance.AllTrades.Count + "; From " + Performance.AllTrades[0].Entry.Time.ToString() + "; To " + Performance.AllTrades[Performance.AllTrades.Count-1].Exit.Time.ToString(), LogLevel.Information);
	if (Performance.AllTrades[0].Entry.Time.Date < (BackTestMode.ToString() == "WalkForward" ? BackTestTo.AddDays(1-TestPeriod) : BackTestFrom))
		Log (ToString() + " ERROR: First Trade @ " + Performance.AllTrades[0].Entry.Time.ToString() + " is prior to start date of " + (BackTestMode.ToString() == "WalkForward" ? BackTestTo.AddDays(1-TestPeriod) : BackTestFrom).ToString("dd/MM/yyyy"), LogLevel.Error);
}
[/SIZE][/FONT]

IMPORTANT NOTE!!! This does not work for the details in the lower pane (including 'Trades' etc.) as viewing these produces another run of the strategy to generate extra trade info, with BackTestMode = BackTest regardless of whether you have selected an optimization run or walk forward run to take a more detailed look at.

At least this way you can guarantee your choice of objective function (i.e. "Optimize on..." selection) and other performance measures in the upper pane are, in fact, correct to the intended dates.

I write rather extensive performance measures to file for all my strategies - so I have a tendency not to bother using these panes anyway.

I would suggest, if it doesn't break any other code, that the NinjaTrader folks correct the BackTestStart variable for the walk forwards and the various user view generated backtests. Then this approach could be simpler and more consistent. Of course, a session template that doesn't leave desired date ranges would be best!


Anyway, I can now perform walk forward analysis on a 24/7 instrument without incorrect trading, and so can you