Discrepancy in Data and Results (BT v. FT)

[CaptainFill]

Hello,

I've been doing some testing on various automated systems and have been encountering a high level of discrepancy between the results of the backtest and market replay.

I am using MarketReplay.net for my replay data and Kinetick for my historical data using continuous contracts for markets such as ES ##-## and CL ##-##.

There appears to be a 2 hour shift in the data when using the global crosshair, I'm not sure if that is the cause of the problem or if the data is simply that much different. Even when running a very simple test strategy such as an SMA cross, there are still significant performance variations.

So I guess my real question is to which data and which results would I expect to be accurate?

Thanks!

[NinjaTrader_Brandon]

Hello CaptainFill,

Thanks for your post.

It is expected to see differences between backtesting, market replay, testing in sim on live data, and live orders.
Testing in market relay or in sim on live data will be the most accurate to a live trade however you would still see differences live as how the fills execute is simulated.
When backtesting you do not get any intrabar fills which will reduce it's accuracy.

The following link can also be helpful for more information about accuracy between real-time and backtesting.

http://www.ninjatrader.com/support/helpGuides/nt7/index.html?discrepancies_real_time_vs_bac.htm

[CaptainFill]

Thank you, Brandon.

I completely understand that the actual fill rates will differ by a varying degree between the replay, live and backtest however I feel that there is still something causing additional errors.

The strategies being tested are calculated on bar close with the executions being triggered at the open of the next bar.

It appears to me that the data is simply different, in particular during the weekend breaks. The real question is what type of data should I rely on as more accurate to emulate live price.

Also, should I not be concerned about the 2 hour difference between replay and simulated?

Thanks!

[NinjaTrader_Brandon]

Hello CaptainFill,

Thanks for your reply.

The Market Replay data we record is unfiltered data. The data you backtest on will be data from your data provider. If you see discrepancies between this data I would need an example and also would need to know who you connect to for data.
As for the time difference, Market Replay data is recorded in eastern time however it should display on your PC as your local PC time. Data from your connection should also be based on your local PC time. You may want to exit NinjaTrader, sync your PC clock, and retest this.
Please use the steps below to change your Time Zone and sync the PC clock, note to set the server to time.nist.gov.

You can sync your PC clock by double clicking on the clock in the lower right corner of your desktop. Press 'Change time zone...' Set the Time zone and press 'OK'.

Once you have done that, click on Internet Time tab set the server to time.nist.gov and then click Update. Your PC clock should now be updated.

[CaptainFill]

Brandon,

No luck on the syncing of computer clock. It is still off. You can see in the image linked below the couple hour difference between the time and the location of the same bar on the other chart.

The replay data used is MarketReplay.net and the simulated backtest is Kinetick.

In the image you can see in red circles the difference in the data. I am assuming that because of the difference in price jumps cause the moving average strategy to show different results.

http://postimg.org/image/5rl5vxb8z/

Thank you!

[NinjaTrader_Brandon]

Hello CaptainFill,

Thanks for your reply.

Do you see the same difference in time if you download the replay data from NinjaTrader rather than MarketReplay.net?

[CaptainFill]

It appears that the NT replay data is the exact same as MarketReplay.net if when I downloaded the data from NT is replaced the data.

It appears that at 17:15 there is a jump to 19:00 in Kinetick. So the question is where is that missing data with Kinetick and why does the replay data have it?

[NinjaTrader_Brandon]

Hello CaptainFill,

Thanks for your reply.

I have verified that both Kinetick and Market Replay are displaying on my PC in my local time, mountain.
Try deleting replay data for the ES 03-15 for yesterday from the Documents\NinjaTrader 7\db\data\20150226 folder
Redownload the ES 03-15 data from yesterday from NinjaTrader and test once more
If it still does not match up write into platformsupport[at]ninjatrader[dot]com on this requesting a call so we can schedule a time to look at this with you.