Can NT cross validate in sample?

[sajosh]

NT is powerful. I'm surprised that NT doesn't offer in sample cross validation. Or I'm surprised that none of my searches (I've searched high and low) revealed this ability or extensions or anyone else wanting it.

Cross validation prevents the optimizer from over fitting curves to data.
Definition of cross validation ... https://en.wikipedia.org/wiki/Cross-...8statistics%29

1) NT can optimize/train on one set of data. 2) During optimization NT should check the fitness (over fitting & under fitting) using a cross validation set of data which is "in sample". 3) Then as a third step, it should apply the parameters to out of sample data that was not used for optimizing or validating.

NT does steps 1 and 3 (room for improvement) but it doesn't seem to do step 2, cross validation. The room for improvement is this manual work should be automated ... "you would need to save your input parameters and re-run the test for the second date range and compare the results" in this thread ...

http://ninjatrader.com/support/forum...+out+of+sample. Also, Walk forward is not the same as cross validation.

With in sample cross validation NT optimizer will avoid over fitting.

Am I overlooking this capability? or
Is there a trivial (not labor intensive) work around for this? or
Is there an addon for crossvalidation? or
Has NT user built one? or

Can NT give guidance on where to start adding it? Perhaps extending @GeneticOptimizer.cs? Would it be possible to code a Performance Type to test half way into the walkforward range to simulate a cross validation test? Then the performance type would sum the peroformance of the optimizer plus the cross validation (half walk forward) to get a total performance?

[NinjaTrader_PatrickH]

Hello sajosh,

Thank you for your post.

Currently Cross-Validation is not used or an option. I am not aware of an add on for this as I would venture to guess it requires creating a new Strategy Analyzer in order to perform this function. The Strategy Analyzer code is not exposed other than the Optimization Types/Fitnesses which would not offer us the needed options for the Cross-Validation.

I will forward this to development as a suggestion and for their insight.

[sajosh]

Hi Patrick,
Thanks for the quick reply.

After years of research, now cross-validation is considered a required part of training/cross/test routines in all forms of data analysis, even trading. Given that, I'm stunned that NT doesn't have cross-validation.

Having Cross-validation will take NT optimizer to another level of reliability.

I've built my own Performance Type. So I'm getting the feeling that there just might be a way to quasi-cross-validate.

The key question is how can Type look into half of the Walkforward range?

NT has some dynamite folks like Bertrand, Josh, et al. So I'm hoping they can get me started with above question.

[NinjaTrader_PatrickH]

Hello sajosh,

I spoke with the product management team on this matter.

Walk Forward Optimization is a cross-validation as it takes the performance of in sample and tests it over an unknown or test period set of data. This would be true to the cross-validation definition.

Is there an aspect of Walk Forward Optimization that is no performing to your standard for cross-validation?

[sajosh]

Hi Patrick,
Glad you spoke with them.

There is a difference.

Most cross validation is done as a part of the optimization parameter selection process whereas NT optimizes (completely) then when performance metrics are at their best a walk forward (Not rolling but let's just talk about a static WF) is done. After the parameter selection.

Most cross validation is done as part of the optimization ... For the purpose of getting parameter sets that are not absolute best for just in sample but rather best on the cross-validation set. Then after the optimization with cross validation is complete a WF is done.

Can the team talk this over a bit more carefully.

But most important ...
How can Type look into half of the Walkforward range?

Does NT have a function I can call from within Type that can take the parameters that optimizer found and run them on half of my WF range? (For simplicity let's just say WF with only one walk.)

[NinjaTrader_PatrickH]

sajosh,

Product Management reviewed and we forwarded this as a feature request.

[sajosh]

Thanks Patrick.

In the mean time, can your developers think of a way for me to code this in a Performance Type?

How can Type look into half of the Walkforward range?

In Performance Type i'm able to get the optimized parameters. Now how can I run a 'backtest' (to stay consistent with NT terminology) from within Type using the optimized parameters? Is there a function call?

[NinjaTrader_PatrickH]

Hello sajosh,

Thank you for your response.

Only the optimizer type/fitnesses are available to program and what you are looking for would involve more on how the underlying mechanics of the Strategy Analyzer work.

[NinjaTrader_PatrickH]

This suggestion has been given the ID # SFT-1366.