Market Replay vs Back Test results

[GKonheiser]

I am noticing differences in System performance between back testing and Market replay and I can see its due to differences in the data. Is the market replay tick data unfiltered, as I'm pretty sure my historical data is from Kinetick which is also unfiltered so in theory there should be no difference?

[NinjaTrader_Jason]

Hello Gkonheiser,

Market replay data you download is recorded from real-time Zen-Fire data. So it is a different data source than Kinetick and can cause subtle differences.

Please note that differences between backtests and real-time tests (this includes the Market Replay) is expected. See the link below for more information.

http://www.ninjatrader.com/support/helpGuides/nt7/discrepancies_real_time_vs_bac.htm

[GKonheiser]

Thanks Jason,

Would you say that market replay is therefor a more accurate representation of live trading then back testing?

If I were to use the same data source for live trading as the market replay source uses should I expect a very close coloration between market replay and live trading results or would there still be significant errors?

[NinjaTrader_Jason]

Yes, you could say that because the strategy will be calculate intrabar (providing Calculate on bar close = False). Upon backtesting, the strategy cannot be calculated intrabar - only at the end of each price bar.

Yes, it will be close - the strategy will be calculated as per the same real-time data.

[cfp462]

Hey, I am also having pretty significant problems with back testing versus market replay. I have both a Zen Fire connection and CQG, and my problem seems to exist regardless of what data I use.

The issue is that back testing is not even remotely close to Market Replay. Which doesn't make sense because even if I use ZenFire tick data, calculating on bar close = true, my results can still be significantly different then market replay.

Issues I have encountered include (but are not limited to):

Market replay can make almost exactly the opposite trades as back testing

Time of entry and exit of trades are different by the time of base period value

When exit on close = true backtesting will make trades that enter and exit on the same day at exactly the same time and record a significant profit

When back testing a kagi 10 minute base period value versus a kagi 600 second base period value with a MACrossOver strategy CalculateOnBareClose = true; will result in very different results

I have been testing the NT platform for the past month and cannot seem to understand all of these problems. I understand that data will be slightly different but I do not understand why I am having problems quite like this.

Thanks for any help that can be provided

[NinjaTrader_AdamP]

cfp,

Likely you are noticing two separate issues here. One is order management having granularity in Market Replay, but not having it in a backtest. The second is probably that Kagi chart styles are "time-independent", so there are differences on your charts.

In Market-Replay, order management can occur intra-bar as it would on a real-time connection. As such stops and targets could be filled intra-bar even if the strategy is operating tick-by-tick. This is because the backtesting engine lacks intra-bar granularity to determine how a bar was built internally. It only has OHLC data available.

You can use this reference sample to backtest tick-by-tick : http://www.ninjatrader.com/support/f...ead.php?t=6652

Kagi bar styles are "time independent".

“Time independence” implies that these bars are not fixed to a set time scale, so for example, a bar “closes” when some other condition not related to time is reached, such as a certain number of ticks occurred, or a certain price range was achieved. Bars that fall into this category are range, Renko, tick, volume, etc.

Time-independent bar styles are mostly built from information received from tick-data, and each bar’s time stamp is only dependent on when the condition for these bars to “close” was reached. As such one bar may take 1 minute 32 seconds to “close”, the next may take 1 minute 14 seconds to “close”.

As a side effect, when reloading this style of chart you may notice differences in the way the bars plot as “time independent” charts are sensitive to their initial starting point. I.e. there may be some missing ticks that would have built the first bar on your chart differently and it cascades through the whole chart.

[cfp462]

Ok so this is starting to make a little more sense, but please note that all my orders are market orders without stops or targets, and calculated on bar close, so the granularity shouldn't really affect my trades.

And to address the issue with Kagi, how do you explain it have exactly the same orders except inverse in back testing versus replay? From my understanding it is forming the bars at the end of the base period. I guess my question is, how can I get these to line up? How can I instruct my strategy to trade live, exactly how it does in back testing?

This is kind of a side note but to your point about charts loading at different times. How do I get my Automated Strategies to load prior data after I enable them so it doesn't take days for them to start trading?

[NinjaTrader_AdamP]

cfp,

For your last question there, you would want to attach them to a chart that has a lot of historical data loaded into them already.

As far as your other question, these bar types can be drastically different because of the "time independence". It also may be something in your code. Likely we will need to see an example of what you are doing to isolate it down further than that.

There is no way to have strategies behave exactly like they do in backtesting but you can make it very close using the following : http://www.ninjatrader.com/support/f...ead.php?t=6652

[cfp462]

Sorry I am still new to all this but, how do you attach a strategy to a chart that has data?

So I ran a test to show you the time difference problem. Both stragies are the SMA strategy provided, ran with 10 and 25 short/long; ran on a 120 second PF chart from 9/23/2012 to 9/28/2012 on the ES 12-12.

Attached are the two trade outputs, and as you can see sometimes the intro time lines up, sometimes the exit lines up but there seems to be no rhyme or reason to the difference. Theoretically the tick data I am using is Zen Fire, and they start on the same data so the chart should be built the same and the should act the same way. Any idea of why this happens?

[NinjaTrader_AdamP]

cfp,

If you right click your chart and go to "Strategies" you can add a strategy to a chart this way by double clicking it, setting up parameters and then setting the enabled field to true.

There will be differences in data between replay and Zenfire if you downloaded replay data. Do you record replay data?

[cfp462]

Why is there differences between replay data, and downloaded tick data if they are both from Zen Fire?

And I have finally found out that all of my issues are simply because I am using Kagi and PF charts. For some reason the back testing does not seem to line up with Market reaplay, and the Market replay has issues, where if I look at the graph it will make trades that are off the lines of the chart.

Is using Kagi and PF a known issue with NinjaTrader? Because I can get my strategies to line up just fine if using Last Minute data, but if I use Kagi Minute data, everything seems to start going wrong.

[NinjaTrader_AdamP]

cfp,

Why is there differences between replay data, and downloaded tick data if they are both from Zen Fire?"

Market replay data is downloaded from our servers, Zen-fire data is different. This would explain a lot of the difference in your P&F and Kagi charts. Different tick data, different bar construction.

P&F and Kagi chart styles are "Time independent" as I have outlined before. We use industry standard formulas for the construction of these bar types. They have inherent draw-backs as well as benefits due to the construction method they use. There is no way to peg P&F bars to a specific time intervals, as they are built up based on price movement.

[cfp462]

So I think I have finally found what the problem is. When back testing on PF charts, it will give you the first X or O of the reversal, when in fact (if your reversal is 3) your point of entry should be on the 3rd X or O. This is the major source of discrepancy between back testing PF and Automating it. Is this a known issue?

Additionally are there any indicator to trade specifically on PF or Kagi charts? I would like to be able to automate buying and selling at reversals, is this possible?

[NinjaTrader_Bertrand]

Hi cfp, yes that would be a known limitation of backtesting those more exotic bars types like PNF, please see our documentation here on this item - http://www.ninjatrader.com/support/h.../bar_types.htm

You could add more granularity to your backtesting as well, helping to enhance it's accuracy.

http://www.ninjatrader.com/support/forum/showthread.php?t=6652

[cfp462]

Is this planning on being fixed anytime soon?

And Is there a way to calculate buys and sells on PF or Kagi indicators?