Backtest is know to be broken - need commitment to fix

[dirkp]

Over the last few weeks I've documented via e-mails to NT support various ways in which the backtest in NT7 is currently broken for Point and Figure charts (I haven't looked much at other chart types yet). Depending on the chart settings these bugs can make any backtest for P&F charts completely misleading.

The problems include:

1) The current backtest uses the high of a new "O" column and the low of a new "X"
as "open" no matter what the reversal and box size settings are. In real-time, NT7
starts the new column correctly once the price has reversed by what's specified in the
box size and reversal settings. The backtest is frequently off by "reversal" x "box
"size"; for a 3-box reversal chart with a box size of 2 that's 12 ticks per round-trip. As
soon as profit targets and stop losses get added to the mix, there can be trades
showing up during backtesting that didn't show up in real time for the same period,
and the other way around. This makes the backtest feature pretty much useless for P&F charts.

2) Sometimes two "O" columns are being drawn next to each other; with P&F charts
that's not supposed to happen. This problem is not limited to the backtest and seems
to show up mostly around trading breaks.

3) The backtest allows to wait for the completion of a specified number of bars before
starting to place trades in order to give the indicators time to get up and running.
Unfortunately it takes this time out of the backtesting period instead of adding it in
front of the backtesting period. This makes it impossible to backtest individual trading
days without having to revert to spreadsheets.

While NT support has by now acknowledged that these bugs are real - and has supposedly put them on the list of things that need to be fixed - so far there is no commitment to a time line for actually fixing these bugs. This is about as useful as no commitment to fix them at all.

That's why I'm posting this message to the support forum now to
- make other traders aware these bugs exist, and
- to get Ninjatrader to commit to a clear time line to get these bugs fixed

Is there anybody else out there interested in getting the backtest to work properly?

[billr]

From my conversations with NT, the backtesting not fuctioning properly is of little interest to them. When I describe the problems I have (In my case with renko charts), I simply get a description of why they don't work. And am referred to the section in the 'manual' that describes the fact that the backtesting has built in logic that does not support many chart types. I suppose that section in the manual allows them to check it off the to-do list??? I am constantly in the process of hiring third party consultants to assist me in workarounds to try to trick the backtesting logic to perform something close to useful. (As you can tell I am not too happy about it) I think the NT support guys are great and help when they can, but frankly I don't think NT as a company really makes much distinction between serious problems that affect real trading vs. tidying up any other software glitches. I get the sense all problems are prioritized from a programmers point of view, not customer/trader point of view. I have been fighting the backtesting for over a year and have had no conversation with NT that suggests they feel a need to change it. Also, I am sure that the majority of amateur traders use time-based charts and not PnF or price movement charts. Unfortunately, those are the forgotten oddities in the NT world. Sorry to be the bearer of bad news.

In the meantime, I have begun a search for an alternate source for a backtesting platform that caters to more serious traders.

[NinjaTrader_John]

Hello Bill and Dirk:
I have read through your posts and verified my replies below with the NinjaTrader engineering team. However before I address these items let me assure you that NinjaTrader takes all user submittals very seriously. We keep a rolling list of all feature enhancements and bugs that get reviewed on a regular basis and we add new features and fix bugs based on what provides the most value to all our clients. Unfortunately we cannot add every new item requested nor can we “fix” items that cannot be fixed such as item 1 below.
Dirk, there is no timeline to "fix" these items since they are either limitations of backtesting a specific chart type or an ideological implementation difference. What I mean by that is we had to choose an implementation direction, ideology, and based on client input at that time we choose the implementation that is currently being used although I understand you would have selected something different.
Sincerely,
John

1. Please consider the nature of backtesting a PnF chart and the underlying inherent differences between real-time and backtesting exacerbated by this particular chart type. In real-time you are able to have your profit targets and stop loss fill based on actual price action inside that PnF column. In a backtest, you are unable to know the sequence of events that happened in that PnF column. It is not possible to pinpoint when your orders will be filled or even at what price as there is not enough granular information. In real-time your orders fill when the exact price action moves to your order’s prices. In a backtest we can only guess what happened based on end of column information. End of column information could contain information from a huge amount of time and the longer the column is you can see how the inaccuracies can become more apparent. This is not a bug and is simply the limitation of backtesting PnF charts. With imperfect information orders filling can be different than what happens in real-life trading.
2. Two “O” columns and two “X” columns can occur at session break. This is the case because NinjaTrader starts every single session fresh. If we did not do this you would have extremely different charts depending on which start date you choose. If you had a 3-days back PnF chart and we did not start every single session fresh, what you saw in today’s columns would look completely different from what you saw of the same day tomorrow. This inconsistency makes for trading incredibly hard and as such we have opted to start fresh charting each session so things don’t change from one day to the next. This is not a bug, but the option to have a choice of whether or not to start fresh sessions is on our feedback list.
3. This is expected behavior. The BarsRequired setting will be taken from the backtest period as no additional dates can be loaded prior to the backtest period without including it into the backtest. I believe we have presented you several options in regards to this already before.

A. Set BarsRequired to 0
B. Program your strategy in a manner that skips the first session of the backtest and run your backtest on two dates. The indicators would have plenty of bars to build up its historical values by the time it rolls into the second session date for your testing when trading commences.

[billr]

Hi John,
I have to admit my tone came across more hostile than I intended. My point was that using special charts such as PnF and Renko places us in an unfavoralbe position with NT (and other platforms as well). From a traders perspective, the lack of functionality using these charts renders NT somewhere between marginally useful to dangerous to use, depending how much you know about NT. On the other hand, my guess is that from NT's point of view the current logic works for the most popular chart types, and changing the logic in place to fix these issues is not seen as critical. I would take that perspective one step further and suggest that programmer/developers see the NT limitations with these charts as known problems with the charts, not the platform. Traders who use them are seen as outliers in effect. Hence, they wont ever be fixed because the problems are not considered problems as such. I just hope these complaints don't lead to NT dropping these charts as a solution.

Thanks.

[dirkp]

Hi John,

It is somewhat surprising that you guys put in a lot of effort into adding P&F, Renko, Kagi and Line Break charts in NT7 - which is great - but appear largely disinterested in putting in a limited additional effort to actually get them to work properly in backtest.

From my perspective, items 1. and 2. are clearly bugs; 3. is an unfortunate implementation choice.

1) While there might be no absolutely perfect way to deal with item 1, there seems to be a reasonably straight-forward option to fix most of the problem. Based on the reversal setting, it is 100% certain which box of a new column was the one that got the first tick of the column. For the most common P&F setting, which is 3-box reversal with a modest box size, just getting the box of the column "open" right would be a huge improvement over the current situation (probably getting rid of 80-90% of the problem). What is more tricky to deal with is the fact that for box sizes of >1 tick the first tick can be anywhere within the box. One way to deal with this is to give the user a way to choose between options like most conservative, most optimistic, % of the box size, and random. This would still not be absolutely perfect, but the residual problem would then be just like slippage. "Cannot be fixed" doesn't really apply here; this seems to be purely a priority issue.

2) If two "X" columns or two "O" columns show up next to each other in a P&F chart, that's a bug, not a feature. Sorry, that's just the way P&F charts work.

3) This is indeed an ideological implementation choice, just a very unfortunate one. There are ways for the user to work around this problem (either via spreadsheets or via work-arounds inside the strategy as you describe below). It's just completely unclear to me what the advantages are supposed to be that the current implementation has over one that adds the waiting period in front of the backtest period, while the disadvantages are very clear.

In terms of seriousness of the problems,
- item 1 makes the current backtest largely useless - if not dangerous - for P&F charts,
- item 2 creates an occasional hiccup,
- item 3 makes the backtest less user friendly than it needed to be

Regards,
Dirk

[Moody1337]

Any follow up on this?

Backtesting on these chart types has been a large issue for us. We need all of these chart types to backtest correctly.

It is a serious bug in the platform that a three line break backtest can look like this simply because the entry/exit of reversals is incorrectly placed:



Thanks

[NinjaTrader_AdamP]

Moody, Dirk, and Bill,

Intrabar granularity and inaccurate fill concerns are present on all platforms in back testing/optimization. Most individuals use back testing/optimization along with walk-forward simulated testing to ensure their strategies are indeed profitable. The main issues are that there is no way to know how exactly how an order would be filled by your broker using historical data, as well as computational complexity issues. For example, imagine running an optimization on an hourly chart of 10 variables over three months with thousands of ticks per hour. Most of the design of the strategy analyzer has this in mind, to be the most accurate it can be while not trading off efficiency, and was not unintended.

There are a couple options for improving accuracy in the strategy analyzer, one is to use a reference sample available in our forums on increasing intra-bar granularity, another is to create your own custom fill types that more accurately reflect the conditions present on your setup, and the third is to do backtesting using Market Replay.

You may find fill types located in Documents / NinjaTrader 7 / bin / custom / type /. The files are @DefaultFillType.cs and @LiberalFillType.cs. You can use these as templates to create a new fill type.

Using Market Replay for back testing would replay historical market tick data, and its possible to have a strategy running and placing trades during the replay on the Replay101 account.

Here is a reference sample on back-testing with intra-bar granularity : http://www.ninjatrader.com/support/f...ead.php?t=6652

Here is some more information on Market Replay : http://www.ninjatrader.com/support/h...ket_replay.htm

PnF charts are time-independent, and as such have inherent weaknesses as well as strengths over time-dependent ones. There are a variable number of ticks per minute, so there is a possible sensitivity to initial conditions. For example, if you start a session at 8:31, the PnF charts will look slightly different in future bars than a chart starting at 8:30. The issue can be compounded over days and days of data, and as such a design decision was made to make the charts less affected by this. It was not an unintended error or an oversight. That being said, your concerns are important to us, and we will definitely take them to heart just as we took the concerns of others to heart when we made our original design decision.

That being said, I do have a suggestion to remedy your issue with our version of PnF charts. NinjaTrader is designed to be very customizable, and as part of this you have access to creating your very own bar types with your own specifications. Should you choose to create your own point-and-figure bar styles, you could make them behave exactly how you want them to behave. The BarType code is located in Documents / NinjaTrader 7 / bin / custom / type / @BarTypes.cs . Please let us know if you would like some direction on where to get started.

Improving the strategy analyzer by having it load more historical data than the testing period in order to get the indicators calculating for the beginning of the testing period is a great suggestion, and we would be happy to forward this to development for consideration. Until then, our best suggestion to you is to increase your back testing time period by a sufficient amount so that the indicators are calculating at the beginning of your desired time-frame.

Our customers are very important to us, and we take all of our customers concerns with equal weight. All of our decisions in implementation and design are meant to satisfy as many people as possible. When we meet constraints, we do our best to surpass them or mitigate them however undoubtedly there is always room for improvement in our software, just like every piece of software in existence. This is why we greatly appreciate your suggestions and insight into improving our platform, and we definitely hear you, and acknowledge and respect your concerns.

Please don't hesitate to contact us should you require additional assistance.

[billr]

Thanks Adam, here are some comments on your response:

"... Most individuals use back testing/optimization along with walk-forward simulated testing to ensure their strategies are indeed profitable." True, but only if the backesting program is sound. Being best it can be isn't necessarily good enough to accomplish what you describe. What good is backtesing/walk-forward if the results do not accurately reflect how the strategy would mechanically respond to the market during that time? [setting aside external issues like broker, data problems. We're not testing that.]

"The main issues are that there is no way to know how exactly how an order would be filled by your broker using historical data," This is factually true, but I contend not a 'main issue' for traders when backtesting. As a matter of fact, I agrue that a perfect fill model is the most useful assumption for testing. An individual has many methods available to account for broker execution issues. A trader can run into this even in discretionary trading. I think this is an invalid justification for lowering the accuracy threshold. Remember, the idea of backtesting is not to predict the future by trying to reproduce the market environment in its entirety, it is to test the merits of a trading model/strategy using the actual, historical price movement of the market as a testing ground.

"...as well as computational complexity issues. For example, imagine running an optimization on an hourly chart of 10 variables over three months with thousands of ticks per hour. Most of the design of the strategy analyzer has this in mind, to be the most accurate it can be while not trading off efficiency, and was not unintended."
If I understand these couple of sentences, herein lies the problem. "... most accuurate...while not trading off efficiency..."

I think you're saying that Ninja has made an intentional decision to trade away some accuracy for efficiency. Futhermore, this trade-off is made more acceptable due to the uncertainty of broker executions in live trading. If this is in fact what you are saying, you are making my case. The product design criteria is programmer centric, not customer/trader centric.

I doubt any serious backtester would prefer quick, efficient results over accurate results. Mainly because the trade off you are describing corrupts any meaningful results. A strategy that is tested in a way that correctly times the entries/exits of trading against historic data and does so using perfect trade execution may not be realistic, HOWEVER it is fundamentally accurate. The perfect trading executions that would not account for broker problems, data feed problems, or slippage are problems that can be managed by the backtester/trader. They are manageable because they are problems systemic to trading and should be addressed independently of how a strategy would trade in a 'perfect' environment. Results that include inaccurate and sometimes impossible trading scenarios (peeking forward) do NOT represent an accurate test of how a strategy would mechanically trade against a given data set for a given period of time, and is not manageable.

If the computational demand for accurately 'running' the historical data is a major problem, I suggest you consider backtesting as a separate platform. I would actually prefer to have it separate from my trading platform. Why share the resources? Historic data can be downloaded in off hours, computational loads could be done during the day without endangering your trading platform. NT could charge separately. Win win.

"... the third is to do backtesting using Market Replay." Market Replay is all I use now. It is maddening how tedious the data download process is, but the accuracy is worth it. That said, I assume it does accurately trade strategies based on the flow of ticks. Strategies mechanically reacting just like they would in a live market (less broker, data issues). Correct?

Don't mean to be too hard on you Adam. I'm sure many share my appreciation for the fact that you are responding.

Thanks.

[NinjaTrader_AdamP]

billr,

It is possible to get tick level intra-bar granularity, which is the main thing affecting the discrepancies between back testing and live trading. Here is a reference sample.

http://www.ninjatrader.com/support/forum/showthread.php?t=6652

The industry standard right now is to approximate trading in back-testing using lower resolution data. I can see your point as to why having more options would be useful, however with the reference sample I provided you should be able to get the highest resolution possible for your back-testing. Please also keep in mind many data providers only offer something like 3-6 months of historical tick data so if you are doing long term back-testing, you need to approximate live trading with lower resolution data at some point.

Our platform is customizable enough you should be able to modify most of the things you and the others have mentioned to your/their specifications. I provided a few starting points in my prior post related to creating your own bar types, as well as custom fill types.

Thank you for your suggestions, we do take them seriously. I will post a tracking number for your suggestion here once I get it back from development. We use these to keep track of peoples suggestions for future releases.

[NinjaTrader_AdamP]

The suggestion for intra-bar granularity options in back-testing has been submitted with tracking number 663.

Please let us know if you require additional assistance.

[billr]

AdamP,
Thanks for your response. If my understanding is correct, the main problems I see with the backtesting is 1) the way the chart is created using historic data and then 2) the limitations of how the strategy is run against that chart. (please note that much of my experience with Strat Analyzer is with range and renko charts) Also, I will describe it how I interpret what is going on. Please let me know if I am mistaken.

1) The backtesting starts by building a chart using historic data. The logic in how the chart is created is different than a chart built in proper sequencing of a live market. The result is that
charts have an inherent smoothness that doesn't reflect live market conditions for that same period. The program appears to have some 'data peeking' forward that builds a chart more efficiently than how the live trading would. i.e. it takes some of the curves out of the road since it already knows where its going.

2) Even with intra bar granularity, the proper sequence of price movement is driven by an algorithm and not the actual time/price sequence that occured live. e.g. if a bar happens to have a large range that encompasses both a stoplimit and profit target, the algorithm will decide which gets hit first, not the actual historical sequence of price movement.

3) Strategy Analyzer assumes close[1] = open[0]. This is a problem for range charts and a deal killer for renko. (this can be solved with some custom programming)

4) Renko charts drop all price movement outside closed bar. It would be as if a normal time candle had the wicks removed and not considered in the backtesting. (this too can be handled pretty easily with third party programs)

While 3&4 are somewhat manageable, items 1 and 2 are the elements that have prevented me from using Strat Analyzer. When you run a strategy and look at the chart, it is easy to see many trades that compromise the calculations. This is why I use market replay.

I hope that Market Replay is as accurate as I am assuming. Even if the process does create some smoothing vs. live trading, the calculations for running strategies in Market Replay appears sound.

Looking forward to comments on my assumptions. Thanks.

[NinjaTrader_Brett]

Assumptions are correct. However if you build the chart with a 1 tick input series this will generate a chart that will be the same as if you were watching it generate live.

-Brett

[billr]

Assumptions are correct. However if you build the chart with a 1 tick input series this will generate a chart that will be the same as if you were watching it generate live.

Hi Brett,
The problem is if you don't want to test against a 1 tick chart. Is there some way to force the chart to be built from a 1 tick input but create another type of chart? (e.g. 5min or 1000tick or 15 tick renko)

Thanks.

[NinjaTrader_Brett]

You can do this however it would take custom coding to bring this secondary series in for finer granularity.

http://www.ninjatrader.com/support/forum/showthread.php?t=6652

[billr]

Brett,
Was I also correct in my description of the Market Replay accuracy?
Thanks.