Different results for backtests with same parameters

[rafe0304]

I sometimes get completely different results for repeated backtests with exactly the same parameters. How can that be?

[NinjaTrader_JessicaP]

Hello Rafe, and thank you for your question.

To help me narrow down what could be occurring for you, I would like to ask, are you able to reproduce inconsistent results with the built-in SampleMACrossover strategy? If you are, please let us know the following pieces of information. This will give us a complete picture of what is occurring and enable us to provide the best assistance possible :

• Screen shots of your Strategy Analyzer settings (instructions below) , including futures expiration date
• Which instrument you are testing
• Which date range you are testing

To send a screenshot with Windows 7 or newer I would recommend using Window's Snipping Tool.
Click here for instructions
Alternatively to send a screenshot press Alt + PRINT SCREEN to take a screenshot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save as a jpeg file and send the file as an attachment.
Click here for detailed instruction

[rafe0304]

No, with the MACrossover benchmark I get consistent results (using the same time frame and the same instrument).

Could it be that with more complicated code, the strategy analyzer sometimes returns early? Because I also get completely different running times.

[NinjaTrader_JessicaP]

Hello rafe0304, and thank you for providing this additional information.

By far the most likely reason your strategy is backtesting inconsistently, is likely due to historical fill processing. I am including a link to the help guide documentation.

https://ninjatrader.com/support/helpGuides/nt7/backtest_a_strategy.htm

In particular, I would recommend reviewing the Fill Type and Slippage settings. Please also note that Ninja will not have any intra-bar data when back testing, and also that your Price Based On (e.g. Last) is restricted to what you set in properties. Finally, you may have some luck generating consistency by setting EntriesPerDirection to 1, and EntryHandling to UniqueEntries .

Please let us know if there are any other ways we can help.

[rafe0304]

Hi Jessica,

Thanks for your quick response.

I can`t see how that:

In particular, I would recommend reviewing the Fill Type and Slippage settings. Please also note that Ninja will not have any intra-bar data when back testing, and also that your Price Based On (e.g. Last) is restricted to what you set in properties. .

would make a difference between backtests, when I use the same settings for each backtest. Upon first sight, this would rather be an explanation for incosistencies between backtests and life tests.

Can you shed some light on that?

[NinjaTrader_JessicaP]

Hi Jessica,

Thanks for your quick response.

I can`t see how that:



would make a difference between backtests, when I use the same settings for each backtest. Upon first sight, this would rather be an explanation for incosistencies between backtests and life tests.

Can you shed some light on that?

Hello rafe0304,

My apologies, you are correct, this would indeed explain a difference between a real life test and a backtest.

One thing that I have found can cause differences between backtests, is inconsistent data being applied. Evey if you select the same data range, to ensure your data starts and stops at the exact same time, please apply an explicit session template, and let us know if this does not bring your data closer together.

[rafe0304]

It seems to work with the session template "CME US Index Futures ETH" . So far , I have used "instrument settings" - the instrument is the e-mini s&p 500.

Thanks for the hint. Is there an explanation for this?

[NinjaTrader_JessicaP]

I am glad to hear that you are receiving consistent results.

Session templates ensure that data begins and ends on the exact same tick. With the single exception of 1 tick bars, every bar in NinjaTrader is built off multiple ticks. So a single tick's difference will mean that every bar that is created afterward will also be different, and this in turn will lead to differing behavior. Different data in to your strategy, in other words, will cause your strategy to behave differently.

Please let us know if there are any other ways we may help.