Guidelines for backtest accuracy

[ds1111]

I'm trying to run a series of backtests on few instruments mostly around 6 range bars. For the sake of discussion just think about a Moving Average Crossover for example as my strategy to go long/short (btw, I'm using COBC = true to make things even simpler).

I'm very familiar with the topic in the NinjaTrader guide: Discrepancies: Real-Time vs Backtest (so please don't point me to the link), but the results with Live SIM and Backtest are so incredibly different that makes all backtest/optimization/walk forward truly useless.

Has someone developed any technique or methodology to help make the NT backtest feature to provide data at least slightly closer to Live Sim in order to obtain a proper understanding of a strategy performance?

How other people are using the backtest funcionality? There must have someone out there that figured out 'better ways' to use Backtest. Can you tell me how you use backtest and how you are making any valuable conclusion on the results?

Thanks for any help!

[koganam]

I'm trying to run a series of backtests on few instruments mostly around 6 range bars. For the sake of discussion just think about a Moving Average Crossover for example as my strategy to go long/short (btw, I'm using COBC = true to make things even simpler).

I'm very familiar with the topic in the NinjaTrader guide: Discrepancies: Real-Time vs Backtest (so please don't point me to the link), but the results with Live SIM and Backtest are so incredibly different that makes all backtest/optimization/walk forward truly useless.

Has someone developed any technique or methodology to help make the NT backtest feature to provide data at least slightly closer to Live Sim in order to obtain a proper understanding of a strategy performance?

How other people are using the backtest funcionality? There must have someone out there that figured out 'better ways' to use Backtest. Can you tell me how you use backtest and how you are making any valuable conclusion on the results?

Thanks for any help!

Backtest is curve-fitting. Its only useful purpose is to eliminate strategies that are a dud. Essentially, if one cannot find optimization parameters using Backtest, then there is no point in doing a forward test, as the strategy has proven to not be viable even with perfect knowledge of the future.

To expect that a forward test will match the results of a Backtest is asking to go broke. A Walk-forward test might be able to provide a tentative methodology/schedule for reoptimization. One can then use that schedule in a forward test, either with small size (the absolute minimum that can be used, given the strategy rules), or in SIM.

Here is a link to another discussion that we had on this issue a while back: http://www.ninjatrader.com/support/f...ad.php?t=43841

[ds1111]

Thanks for the reply. I appreciate the discussion about expectancy, etc... but the issue is that backtest data seems not to be reliable at all. So, any conclusion you may want to make is flawed from the start. I have strategies that score a Profit Factor of 3.2 and are losers when running against live data.

Put a MA Crossover strategy with EnterLong/EnterShort, and look at the backtest charts, even the minimal concern with the auction process filling at the ask for the longs (and bid for the shorts) is ignored. So, I'm not sure what I can call a backtest that ignores the spread... and then you try to optimize / walk forward this... and of course, garbage in/garbage out... and you are just wasting your time.

It's hard to believe I'm the first one running into this, but what I would like to see is this thread to evolve into a discussion of 'best practices' for making the backtest at least "usable".
One idea I had is to create a 'backtest' Boolean in my strategy, and then code a framework to at least 'mimic' a real life auction process when running backtest to fix the most blatant weaknesses of the backtest implementation in NT (e.g. a proper implementation of slippage on a Market Order).

So, let's step back and let me rephrase my questions to anyone out there:
1- How are you using the backtest/optimization/walk forward information in a reliable way for evaluating day trading strategies in the NT platform?
2- What steps are you following to evaluate a strategy? (e.g. are you ignoring backtest, and using Market Replay and Live Sim? if using backtest, how are you using it?)
3- How do you compare your Market Replay with Walk Forward? Do they match?
4- Can you give some examples in which your strategy backtest matched what you saw in Live Sim or Market Replay?

Again, any discussion around this would be extremely helpful to this community. I don't want judge anybody's procedures to do this. I'm truly trying to gather some 'pragmatic' lessons, before I start to code my own backtest capability!

Thanks!

[koganam]

Thanks for the reply. I appreciate the discussion about expectancy, etc... but the issue is that backtest data seems not to be reliable at all. So, any conclusion you may want to make is flawed from the start. I have strategies that score a Profit Factor of 3.2 and are losers when running against live data. - Harldy surprising, unfortunately. A backtest only tells you what happened in the past, optimized to what is essentially prescience of the future, as backtest is the only time that we know exactly what happened after we entered the trade, and we can adjust parameters to make the aggregate of trades look viable. All that shows is that we have a strategy which can probably be viable. You would have to then validate that in the past, using optimization over a period did in fact lead to positive results in some next period. That would be a walk-forward test, as you are aware.

By the way, on what kind of bars are you testing? In my opinion, if you are not using Range bars, your results may be non-predictive in the first place. Range bars are the only bars that are actually representative of market action. You make or lose money whenever a range is completed, and it is independent of time. The motion of "time bars" is apropos of almost nothing, as the passage of that time does not correlate to any specific profits/losses. Longer time bars do ameliorate this problem somewhat: but short time frame bars are almost an exercise in randomness.

Put a MA Crossover strategy with EnterLong/EnterShort, and look at the backtest charts, even the minimal concern with the auction process filling at the ask for the longs (and bid for the shorts) is ignored. So, I'm not sure what I can call a backtest that ignores the spread... and then you try to optimize / walk forward this... and of course, garbage in/garbage out... and you are just wasting your time. - This one is a matter of "point of view", but NT does actually use slippage of 1 tick on Market Orders, and you can specify the amount of slippage if you want. It is right in the Strategy Analyzer interface. If you look at the picture that I posted you will see in fact that the entires are 1 tick beyond the extreme of the preceding bar, (entry was a Market Order at the Close of the bar). More normally, I use Limit Orders, and they do have a slippage of zero off the Last Price. As I said, it is a matter of preference. As I only trade instuments that have a 1 tick bid/ask spread, as far as I care, the real price is the last price that someone actually paid. (In point of fact, when I place manual orders, I place opening buys at the bid, opening sells at the ask, and wait for the market to tick back and put me in). If the market takes off without me, no sweat. Trades are like buses: there will be another along soon enough.

It's hard to believe I'm the first one running into this, but what I would like to see is this thread to evolve into a discussion of 'best practices' for making the backtest at least "usable".
One idea I had is to create a 'backtest' Boolean in my strategy, and then code a framework to at least 'mimic' a real life auction process when running backtest to fix the most blatant weaknesses of the backtest implementation in NT (e.g. a proper implementation of slippage on a Market Order). - I find the implementation to be pretty well done. One just has to figure out how to use it. Most of my initial work consists of using the optimizer to find a viable timeframe that shows a profit. From there it is pretty manual trying to see how previous period parameters work into the future, until I think I have something viable, then I see if it can be automated as a walk-forward test, using the schedule that I have found. Then more tweaking of the schedule. Then the Market Replay. Next comes manual, or automated trading in SIM. Finally ...

So, let's step back and let me rephrase my questions to anyone out there:
1- How are you using the backtest/optimization/walk forward information in a reliable way for evaluating day trading strategies in the NT platform?
2- What steps are you following to evaluate a strategy? (e.g. are you ignoring backtest, and using Market Replay and Live Sim? if using backtest, how are you using it?)
3- How do you compare your Market Replay with Walk Forward? Do they match? - Pretty much, though not always exactly, especially if using Limit Orders, as there are fill variations..
4- Can you give some examples in which your strategy backtest matched what you saw in Live Sim or Market Replay? - The trades in the pictures look the same to me.

Again, any discussion around this would be extremely helpful to this community. I don't want judge anybody's procedures to do this. I'm truly trying to gather some 'pragmatic' lessons, before I start to code my own backtest capability!

Thanks!

To answer all your questions would be quite a treatise in itself. In fact, our optimization/backtest/validation procedure manual runs to quite a big read, even if one ignores the chapters that explain the methodology and just goes on to the cookbook parts.

I have addressed a few of your issues within the quoted text of your comment.

[ds1111]

Thanks a lot again for a comprehensive reply. That's the type of discussion I'm interested, and I may learn something in the process. I will put some examples over the weekend or next Monday as a response. Maybe you can help me understand the discrepancies. Have a great weekend.

PS: you mention a manual ("In fact, our optimization/backtest/validation procedure manual runs to quite a big read, even if one ignores the chapters that explain the methodology and just goes on to the cookbook parts"). Which manual are you referring to, and where can I have access (if so) to it?

[koganam]

Thanks a lot again for a comprehensive reply. That's the type of discussion I'm interested, and I may learn something in the process. I will put some examples over the weekend or next Monday as a response. Maybe you can help me understand the discrepancies. Have a great weekend.

PS: you mention a manual ("In fact, our optimization/backtest/validation procedure manual runs to quite a big read, even if one ignores the chapters that explain the methodology and just goes on to the cookbook parts"). Which manual are you referring to, and where can I have access (if so) to it?

The manual is the one that we use internally for validating and testing systems. It is not available for reference or sale outside of our group. At least not at this time.

If you use your Google-Fu, you should be able to find a few tomes about system optimization and backtesting.

[ds1111]

Ok... after spending hours and hours researching this. I found a couple of interesting things. Sorry for the delay btw, but I'm still working on this journey of understanding better the backtest feature and how to make the most of it.

I realized that the fills as related in my previous post was incorrect because I'm using a modified synthetic version of Renko bars (and therefore my statement about bid/ask fills on Market Orders as you pointed out was also incorrect ) . So the opens of the bars are virtual prices and was causing some problems including with the 'enhanced' backtest results I was obtaining.

I found in one of the forums in the internet that someone using Renko or synthetic Renko's should use the following line of code to skip 'gaps' (within OnBarUpdate):

if (Volume[0] == 0 || Volume[1] == 0) return;

It does causes the backtest to produce different results. In general with my strategy using the bar volume check above, the PF and Performance (in $) increases, and the "Percent Profitable" decreases. So, I'm still trying to learn how this is happening and why. Interestingly enough, even using regular NT native Range bars the same behavior happens when using the bar volume check (i.e., PF decreases and PP increases). I would expect that with Native Range bars, bars with volume = 0 wouldn't be painted and I was a bit puzzled that the code statement above would produce different results (so, something else for me to investigate).

In any way, I will continue my pursuit for a better understanding of the backtest engine as it will be key for me considering the complexity of my strategy and running in MR (or live SIM) all possibilities would be close to impossible.

If anybody has learned something that could add to this thread it would be greatly appreciated. I'm also looking for any papers or articles about the NT backtest engine, or the backtest/optimization process (outside of what is available in the NT help site of course). Some pragmatic experiences, like the simple Volume check above or any other similar solution that you've found helped to produce better results when backtesting would also be invaluable.

Thanks again.

[koganam]

Ok... after spending hours and hours researching this. I found a couple of interesting things. Sorry for the delay btw, but I'm still working on this journey of understanding better the backtest feature and how to make the most of it.

I realized that the fills as related in my previous post was incorrect because I'm using a modified synthetic version of Renko bars (and therefore my statement about bid/ask fills on Market Orders as you pointed out was also incorrect ) . So the opens of the bars are virtual prices and was causing some problems including with the 'enhanced' backtest results I was obtaining.

I found in one of the forums in the internet that someone using Renko or synthetic Renko's should use the following line of code to skip 'gaps' (within OnBarUpdate):

if (Volume[0] == 0 || Volume[1] == 0) return;

It does causes the backtest to produce different results. In general with my strategy using the bar volume check above, the PF and Performance (in $) increases, and the "Percent Profitable" decreases. So, I'm still trying to learn how this is happening and why. Interestingly enough, even using regular NT native Range bars the same behavior happens when using the bar volume check (i.e., PF decreases and PP increases). I would expect that with Native Range bars, bars with volume = 0 wouldn't be painted and I was a bit puzzled that the code statement above would produce different results (so, something else for me to investigate).

In any way, I will continue my pursuit for a better understanding of the backtest engine as it will be key for me considering the complexity of my strategy and running in MR (or live SIM) all possibilities would be close to impossible.

If anybody has learned something that could add to this thread it would be greatly appreciated. I'm also looking for any papers or articles about the NT backtest engine, or the backtest/optimization process (outside of what is available in the NT help site of course). Some pragmatic experiences, like the simple Volume check above or any other similar solution that you've found helped to produce better results when backtesting would also be invaluable.

Thanks again.

As a start, I would strongly suggest that you never test anything using Renko Bars. We have discussed this quite a few times on the forum. Here is the latest: http://www.ninjatrader.com/support/f...ighlight=renko

[ds1111]

NT support says that backtest would work if we use a 1t series to execute on (http://www.ninjatrader.com/support/f...ead.php?t=6652) . Do you know if this is true for Renko or other types of Renko bars (e.g. UniRenko)?

Any comments on using the Volume[0]/[1] check? are they of any value?


Thanks!