Confused about backtesting instrument data

[risingfire]

I have imported one month of historical data (June 2014) for the ES 09-14 contract. However, I'm confused about what I am seeing in the backtesting chart. It seems to be pulling data from time periods that I have not loaded and does not exist. When I open the historical instrument manager, I only see data for June 2014. However, backtesting shows data from 1/2/2014 to 5/2/2014. Then it gaps to 6/12/2014.

So two issues:

1. Where is it getting this non-existent historical data?
2. Why is it gapping to 6/12/2014? I thought I would see data from 6/2 to 6/27/2014.

I have attached a screenshot illustrating the issue.

[NinjaTrader_PatrickH]

Hello risingfire,

Thank you for your post.

The historical data that it is using is likely from one of the previous contracts. NinjaTrader will merge the historical data for futures contracts to create a continuous stream of data from one contract to another. This is controlled by the Merge Policy in NinjaTrader under Tools > Options > Data. You can find information on the Merge Policies at the following link: http://www.ninjatrader.com/support/h...rical_data.htm

It is not clear why it is jumping to June 12th in your historical data. This could be due to the roll over dates in NinjaTrader for the ES contracts. To quickly test for this you can go to Tools > Options > Data > Merge Policy and set the Merge Policy to DoNotMerge. Then run your backtest once again and advise if the historical data is still jumping.

[risingfire]

Thanks, Patrick. Part of the issue was with the merge policy. I had it set to "MergeBackAdjusted". Setting it to "DoNotMerge" solved the data gap problem, as well as picking up data that I did not have historical information.

But I'm still a bit confused about why, even with MergeBackAdjusted, it was picking up price data that I did not historically import. I *do* have Market Reply data for that time period (that is greater than my historical import), but I thought that Strategy Backtesting only used historical data, not market replay data. Is that correct? Is so, how odd that the backtesting was picking up that price data.

But, even after setting the merge policy to DoNotMerge, the backtest was still not loading all of June's price data. So I deleted the historical price data and re-imported. This seems to have solved that issue as well.

[NinjaTrader_PatrickH]

Hello risingfire,

Thank you for your response.

I am glad you were able to resolve this matter. For the merge of the data you did not import, it is likely from a previous contract that it was merging into the current contract.

[risingfire]

Well, still having an issue afterall. I'm not sure if this is related to our conversation or something completely different. But it is something that I have noticed before with market replay data, but could not reproduce it until now.

It seems like there are times that the price data is completely erroneous on the transition from one day to the next (midnight). At least I do not think this behavior is happening at other times.

I have attached two charts show this issue. I am connected to Market Replay and set the date to 6/25/2014 11:50pm. I load a chart with ES 09-14. All the data is fine until 6/24 at midnight, at which point the price data is simply incorrect from 6/25 0:01 and forward. (See the first chart.)

Then I set the market reply date to 6/26/2014 11:50pm (exactly one day later). Now the market replay data is correct. (See the second chart.)

The merge policy is DoNotMerge. So it really should just be showing the 09-14 contract.

What could be causing this behavior?

[risingfire]

Here it just happened again. I set the date to 6/27 and price data after midnight 6/27 is incorrect. I did delete all the market replay files in the data directory and reloaded them using the Market Replay Download utility. So I do not believe it is an actual data issue. Am I just missing some configuration setting somewhere?

[NinjaTrader_PatrickH]

Hello risingfire,

Thank you for your posts on this item.

There is no setting you should be missing. Can you send me your database so I may test this on my end? You can do this by going to the Control Center-> Help-> Mail to Support-> ensure the database option is enabled. Please reference this thread in the subject line: http://www.ninjatrader.com/support/f...ad.php?t=67361
And please place 'ATTN: Patrick - 1108909' in the body of the e-mail.

[risingfire]

Thanks, Patrick. Just sent the email and database, so should be available to you soon.

[NinjaTrader_PatrickH]

Hello RisingFire,

Thank you for your patience.

This is due to the historical data in your database being loaded versus the MArket Replay data. If you have historical data in your database NinjaTrader will use it to backfill the data on the chart when using Market Replay. However, if we play through let's say June 24 to 26 the data we see on the chart is from the Market Replay data. So at that point if we Go To a time ahead, NinjaTrader will backfill the data form the historical database and thus the data will be different. Now that you completely removed the database, there is no historical data to backfill the charts with and now the data is blank unless replayed.

Please let me know if you have any questions.