backtesting renko ranges ?

[benrock]

is it true rthat results will be completly false with these 2 time frames on a back test ? if so why would you allow it to run ?

[NinjaTrader_PatrickH]

Hello benrock,

Thank you for your post.

Backtesting on Renko and Range bar types can return results that would not reflect what would happen in the real-time market (tick by tick). Backtests are ran on Historical Data, which is just the bar that printed not each tick that made up that bar. So for example; you have an 8 tick stop loss set, and you run a backtest on a 10 range bar type. Even if the market had moved down to your stop loss it would not fill because you are viewing only the bar (the 10 range in this case).

We recommend using a smaller interval type for granularity in your backtesting. A great reference sample on this can be found on our support forum at the following link: http://www.ninjatrader.com/support/f...ead.php?t=6652

For information on the discrepancies between real-time versus backtest please visit the following link: http://www.ninjatrader.com/support/h...ime_vs_bac.htm

Please let me know if I may be of further assistance.

[benrock]

ok so why are they there is they give wrong results ?

and if my strategy is for 8renko , then what is best way to back test ?

[NinjaTrader_Bertrand]

benrock, the most accurate way to access a Renko strategy performance would be using our Market Replay feature -

http://www.ninjatrader.com/support/helpGuides/nt7/market_replay.htm

[cesario]

When running a backtest on a Renco chart, erroneous entries occur when the bar following the entry bar is of opposite direction. For example, if program a strategy to enter on the close of a green bar, and the bar following the green entry bar is a red one, then the strategy analyzer will take the entry as being on the red bar and not the green bar.

The cause of this is that ninjatrader cannot deal with the particularities of Renco bar charts: The Closing level of a bar is not necessarily the Opening level of the next one. This is very straight forward with Renko testing, anyone working with Renko bars know about this, and plan development around this.

I am at awe that ninjatrader has dropped the ball on this. The explanation that “ninjatrader is an event driven language… and so there is no on-bar-close event” that is circulating in another post is not really satisfying.

The first action t fix this is to acknowledge that a mayor bug exists in the application. The problem can be easily replicated; it can be easily fixed too. If your team does not know how, I’ll be happy to point them in the right direction.

Please acknowledge that there is problem with Renko bar testing and fix it. Instead of telling your customers of ways to go around it that do not really work and do not really address the issue.

Looking at your website, at all the marketing done by your company, having myself purchased a license, seeing how professional you guys are, and then encountering this flaw and your refusal to even acknowledge it is concerning. Perhaps the application is not as good as you portrait it to be? This is causing me – and others I bet – to look at other platforms that can work with Renko bars.

A serious, professional, and informative answer about this issue is greatly needed.

[NinjaTrader_PatrickH]

Hello cesario,

Thank you for your post.

Addressing your example here; backtesting is always CalculateOnBarClose = True, this will result, as in your example, with the entry on the open of the next bar rather than the close of the previous bar as we believe this allows for a more accurate opening of the position.

As Historical Bars do not include every tick that built that bar but contain the Open, High, Low and Close of the bar, using either intrabar granularity or Market Replay is not a work around but a viable solution to the backtest that the user wishes to use.

Please let me know if I may be of further assistance.

[marcow]

backtesting is always CalculateOnBarClose = True,

I really think this currently is the biggest drawdown in NT.
Why is this ? Please let me know, because I really don't get this.

Aren't today's computers powerfull enough to backttest on tick level ? I think they are.
I render 3D animations with complex lighting/shadow/reflections/refractions in a few hours, and NT isn't capable of backtesting 1 month of tick data in that time ? come on... ( I do realize NT backtesting has to be processed in chronological order, and most 3D rendering is not, but still...)


Why is NT giving false backtest results, and are we doomed to use market replay and wait forever for the results because speed is limited to only 500X ?
I'd rather wait 1 hour and have correct backtest results, then have incorrect results in 5 min.

Is this some sort of fake "optimization" so that NT looks and feels faster ?
Please NT, make backtest tick-based, there are no excuses not to !
We want correct backtest results !
forum members, please reply if you agree...



by the way, If NT was using tick-level backtest, Renko and Range charts results would also be fine.

[NinjaTrader_PatrickH]

Hello marcow,

Thank you for your post.

Backtests are ran on historical data which is downloaded from your data provider, and backtesting can be performed on any historical data you wish, including tick data.
Let's say you wish to run a backtest on a 15 minute chart; each bar is the Open, High, Low and Close of 15 minutes and not each tick that constructed that bar. This is the nature of historical data, this cannot be changed no matter what data provider you choose to use.

This is why we recommend using intrabar granularity to add a secondary bar series to your strategy. In this manner you can use a granular secondary bar series to achieve an "intrabar" fill. You can view our reference sample on this item at the following link: http://www.ninjatrader.com/support/f...ead.php?t=6652

If you wish to use a secondary bar series and use tick data but in the case your data provider does not support historical tick data or you do not have access to historical tick data for the instrument you wish to backtest then you will not be able to run a backtest for tick data. The availability of historical data from your data provider is not determined by NinjaTrader. Therefore making backtesting tick based will alienate those without access to historical tick data.
For information on the data provided by your connectivity provider please visit the following link: http://www.ninjatrader.com/support/h...rical_data.htm

Market Replay only goes to 500 times the normal speed, this is correct. However, Market Replay is designed to replay the market which can be used to test your strategies, but using the Startegy Analyzer will allow you to run historical analysis of your strategy and provide you with performance results as well. For information on the Strategy Analyzer please visit the following link: http://www.ninjatrader.com/support/h...y_analyzer.htm

Please let me know if I may be of further assistance.

[cesario]

Patrick,
Thanks for your prompt reply. CalculateOnBarClose = True is not the issue here. Please see the attached. I want the strategy to enter at the close of the second red bar, but the strategy analyzer is unable to do that. Instead it enters way off on the next bar. This skews the testing results, as you can see in the figure: that particular trade would have doubled.
The problem is the perceived “gap” on Renko bars while looking at historical data. The way to solve this is to force Entries on Renko bars to be exactly at the close of the entry bar, and not at the opening of the next bar. That is what is going to fix this issue. I’m sure some logic can be added to make this happen on Renko bars backtesting. I’m sure your engineering teams can handle this. There are many other examples I can give you if you are interested in showing them to your quality assurance team.
Using intrabar granularity is very demanding on system resources and a mess to test and debug. But that is the way it has to be done now. Still, it becomes hard to trust the results since there are obvious issues with other parts of the application. And how can you replay 5 years of data on Market Replay? Last time I checked you had to download it one day at a time.
If straight forward Renko charts backtesting was beyond the capabilities on Ninjatrader, your engineering team would not had added that feature to the Strategy Analyzer. Please take this as free information about an issue with the Strategy Analyzer and disseminate it to your colleagues so someone can take the appropriate action to fix this bug.
I hope you can be part of the solution to this issue.
Cesar.

[cesario]

Here is the attachement

Here is the attachement....

[marcow]

hello Patrick,

Let's say you wish to run a backtest on a 15 minute chart; each bar is the Open, High, Low and Close of 15 minutes and not each tick that constructed that bar. This is the nature of historical data, this cannot be changed no matter what data provider you choose to use.

But if I record the data myself from a live connection on a renko chart, I do have tick data, correct ?
Why does NT discard this valuable tick data during backtest and reconstructs it into OHLC data ? Backtesting Renko charts with OHLC will give false results, why is it even possible to backtest Renko if the results are incorrect ? Why not completely block Renko (and Kagi and LineBreak) charts from backtesting ?

Do a rearch on this forum and watch how many members have issues with Renko backtesting.

This is why we recommend using intrabar granularity to add a secondary bar series to your strategy. In this manner you can use a granular secondary bar series to achieve an "intrabar" fill. You can view our reference sample on this item at the following link: http://www.ninjatrader.com/support/f...ead.php?t=6652

I really think this "workaround" should be hardcoded into NT. Backtests on Renko should be done with 1 Tick granularity and nothing else.
Why is it always the customer who has to fix these NT shortcomings ?
Why all these "workarounds" for a problem that shouldn't exist ?

Market Replay only goes to 500 times the normal speed, this is correct. However, Market Replay is designed to replay the market which can be used to test your strategies, but using the Startegy Analyzer will allow you to run historical analysis of your strategy and provide you with performance results as well. For information on the Strategy Analyzer please visit the following link: http://www.ninjatrader.com/support/h...y_analyzer.htm

Problem is that only MarketReplay will give relliable results on Renko charts, backtest does NOT. Performance results from MarketAnalyzer are useless on Renko.

I have created a contiunous 9 month ES future contract which I use in market replay.
If I run the strategy on 500X speed, I have to wait 2 days before I get performance results. Insane ins't it ? Why oh why isn't NT capable of doing this in let's say 1 hour on a 8 core 3GHZ system ?
I mean in 2 days I can 3D render a complete full HD animation.

suggestion: why not create a second MarketAnalyzer that runs really fast on marketreplay data ? And fix the download of market replay so it is possible to download more than 1 day at a time.

The availability of historical data from your data provider is not determined by NinjaTrader. Therefore making backtesting tick based will alienate those without access to historical tick data

.

Then I would suggest to make an option in backtest in which the user can choose to use tick- or OHLC- data for backtest.

If I collect historical tick data myself, I expect NT to use it properly.
Why does NT backtest in OHLC if there is historical Tick data available ?
Backtests on Renko,Kagi and Linebreak should be done on tick level and nothing else. If not possible, then block these from backtesting !

Again, in my opinion the biggest drawdown of NinjaTrader is not being able to backtest on tick level.

Marco

[NinjaTrader_Brett]

Hello,

Thanks for the follow up.

This is expected and the strategy analyzer works correctly as designed which is to run on the inputted bars which in this case is 10 renko bars. Anytime you backtest any Range or Renko type bar just simply due to the nature of the bar there are two problems.

1) As you point out the open of the trade is done on the open of the NEXT bar. This is by design, we feel this gives more accurate backtest results for all other backtest scenario's and we would not change this, however on Renko bars this will result in an entry that is +Renko range from the close of the bar if its a reversal bar.

2) Consider a 10 renko or range bar and you place a stop 5 ticks down from entry. In the real world live market action could go down as much as 9 ticks and return and there would never have been a bar formed as you have to have 10 ticks to form the bar. In this case your stop losses and profit targets have a much higher chance of getting hit in live environment.

Both these issues are resolved by using IntraBar granularity to run your backtest, in fact its the only way to resolve this and get even more accurate backtesting results. You must think about the bar type and what it does; Renko or Range is meant to SMOOTH out the bars, well that directly translates into smoothing out your trade results in backtest as well. So we needed to get that back and how we get that back is by adding intrabar granularity. Please see this sample:

http://www.ninjatrader.com/support/forum/showthread.php?t=6652

Let me know if questions.

[marcow]

hello Brett,

You must think about the bar type and what it does; Renko or Range is meant to SMOOTH out the bars, well that directly translates into smoothing out your trade results in backtest as well.

Here's the million dollar question:
Then why are MarketReplay results different from MarketAnalyzer results ?
Why does MarketReplay trigger intrabar Renko and MarketAnalyzer doesn't ?

Both these issues are resolved by using IntraBar granularity to run your backtest, in fact its the only way to resolve this and get even more accurate backtesting results.

-even more accurate backtesting results should be less inaccurate results.
-Intrabar granularity should be default on backtest.


I would like to post another suggestion to improve the Renko backtest process:
and that is to add wicks to the Renko chart. This way, backtest on Renko not only has Open and Close, but also High and Low, thus triggering intrabar, isn't it ?

[NinjaTrader_Brett]

The concept to understand is as follows:

->Strategy Analyzer runs on Bar Data
->Market Reply runs on Live Data

I will forward the suggestion, I know there is some Better Renko bars floating around that can be downloaded as well if you haven't already.



-Brett

[marcow]

thank you Brett,

The concept to understand is as follows:

->Strategy Analyzer runs on Bar Data
->Market Reply runs on Live Data

Correct me if I'm wrong, but the purpose of the StrategyAnalyser is to get performance results of a strategy in the time period backtested, as if that strategy would have run live during that period.

Because the StrategyAnalyzer runs on bar data, the results will be fine for Minute charts, but will be completely incorrect for Renko,Range,Kagi and Linebreak because there are no intrabar fills. For the latter Bar types I'm forced to use MarketReplay. Which is almost unusable because it takes 24 hours to replay 3 months of data.
And even when replay has finished, using the cached data as backtest gives incorrect results.(why is this ?, is the tick based replay data converted to bar data ? )

Fact of the matter is that NinjaTrader is unable to produce reliable backtest results for Renko,Range,Kagi and Linebreak charts. That's 50% of available chart types.....

Please fix this in future version/updates.

Marco