Market Replay 7 vs. 8 Discrepancies

[iantg]

I have recently started migrating my NT7 strategies over to 8 to start to compare results between market replay on both platforms. From what I have read I am expecting to have similar results in market replay, though I have heard that significant improvements have been made to make the basic back-testing more accurate in 8. From what I have seen published and promoted by NinjaTrader, Market Replay in 7 was generally considered accurate and a good representation of how how live trading in the market would perform. It was my impression that there would not be a significant deviation in market replay results between these two platforms, but nonetheless I am in the process of going through some validations to check some of my strategies against 8 now.

After seeing some initial differences, (some small and some huge), I think I have found a good example of one that has major enough differences that it should be looked at by NinjaTrader support. I recreated the two strategies in both 7 and 8 reasonably well, so that they can be tested side by side to evaluate what is causing the large discrepancies.

It's a very simple test strategy just to see how limit orders are handled specifically on the exit by way of a profit target. Therefore I am using a 1 tick profit target against a greater stop loss, just to see how and when a 1 tick limit order (Profit Target) will not get hit on an exit. Here are my observations: The NT7 strategy has around a 20-1 win to lose ratio while the NT8 strategy has maybe a 10 to 1 or 15 to 1 win to lose ratio.

So I guess my question is how would you explain these differences and which assumptions are more representative of the real market? Are we assuming in 7 that profit targets are always filled on touch? Are we assuming that in 8 all profit targets are only filled on pass-through? Or is it less black and white?

Either way I would like to know the underlying assumptions being made by both 7 and 8 regarding this, so I can understand this behavior.

For testing purposes I used the ES Jan 04-8 2016 on a 250 tick chart in Market Replay.

Please let me know if you need any further information to test on your end.

Thanks,

Ian

[NinjaTrader_ChelseaB]

Hello iantg,

There have been some changes to the real-time simulation engine. I will need to further research specifics about how limit orders fill in real-time.

In the meantime, I am providing a link to a forum post that demonstrates how to find what is causing differences between tests.

https://ninjatrader.com/support/forum/showthread.php?p=514402#post514402

Are you including any historical data with your tests or are you ensuring that no historical data is included?

[iantg]

Hi ChelseaB,

Thank you for following up with me. The test were conducted in Market Replay with no data previously loaded historically that already had trades. In terms of the data itself.... It was replay data download from Kinetic in NT7. In NT8 when I loaded it for the first time it asked me if I wanted to migrate my existing data over, so it converted the original NTM file into a NRD file. But unless there was some degrading of the quality in the conversion process should in theory be the same.

If you run the two scripts I attached in 7 and 8 respectively using the ES 250 tick chart it will produce what I experienced. Around 20 to 1 winner to loser ratio in 7 and around 15 to 1 winner to loser ratio in 8 on the same trades. So this got me thinking that something has dramatically changed in the simulation engine between 7 and 8 though I have never heard anyone mention this anywhere.

I was just hoping to get to the bottom of this and understand the assumptions in 7 and 8. My guess is that this pertains to the topic of touch vs. pass through on exits using profit targets.

Let me know what you find out.

Thanks,

Ian

[iantg]

Any Updates to this question?

Pertaining to Market Replay: What I would specifically like to know is the following:

1. What logic changed in the decision engine for filling entries and exits between 7 and 8 for Market Replay.

2. If it was determined that a major change needed to be made with 8, does this imply that the market replay engine in 7 was flawed? What are the specific areas that were determined to be incorrect, or unrealistic?

3. What would be the steps one would take to reconcile results between 7 and 8. Is this as simple as adding 1 extra tick for entries and exits? What would be needed to walk between 7 and 8?

4. What are the underlying assumptions made by each (7 & 8) with respect to execution speed. Would this be a good proxy for someone in Asia executing local entries / exits to the CME in Chicago, or would this be closer to someone collocating at the CME on a dedicated server? How would you characterize the standard assumptions built into the decision engine on both 7 & 8?

If anyone from NT or any senior members have any insight into this I would greatly appreciate it.

Thanks,

Ian

[sledge]

I downloaded both your files and ran a compare, and saw 1 exit fill missed by NT8. It was during high volatility. NT8 does further refinement beyond seconds. I don't know if the same order is maintained between NT7 (which is only at seconds) and NT8 converted data.

Well, if you ran both replays together and slowed it down at a discrepancy, you'll probably see what's going on.

[iantg]

Thank you sledge for taking the time to compare my two strategies. I have found specific orders that filled differently between the two, and it looks like it is mostly on the exits. In NT 7 there were more exits filled. In NT 8 it missed quite a few profit target exits. It also looked like it hit the market order exits easier in NT 8.

While I can observe this behavior and have my own theories about it, I would really like to understand the rational from NT about why the behavior was changed and then to learn more about the specific rule changes.

For anyone running strategies in Market Replay it would be very helpful to know if some of the underlying assumptions in 7 were less than realistic and they have since corrected them in 8, or if perhaps 7 would be more indicative of faster executions that could be achieved through low latency HFT systems and 8 is more representative of someone clicking from a local machine far away from an exchange.

I think it's obvious that the fill engine has dramatically changed from 7 to 8 but I have yet to hear any explanation as to what assumptions changed, why they made the decision to change it, and if one is considered more realistic to the markets vs. the other.

Any input would be appreciated.

Thanks,

Ian Guthrie

[sledge]

I'm not sure they can explain it.

My strategy takes a couple trades a day depending on conditions - and usually goes through the target without issue as it's way bigger than 1 point target.

I have about 3 trades over 3 years that "miss" in NT8.. considering how I busted their chops through the entire beta and beyond (and some how end up being mostly right), it's something I can deal with.

[NinjaTrader_PatrickH]

Hello iantg,

Thank you for your patience.

This was discussed in the past on the following thread: https://ninjatrader.com/support/foru...ad.php?t=96647

In short, there was logic in place in NinjaTrader 7 that would randomly 'guess' at a starting volume for Market Replay. This guess would make the calculation inefficient in NinjaTrader 7. To improve performance we elected for a more conservative approach. Therefore you will see some orders fill faster in NinjaTrader 7's simulator then you would see in NinjaTrader 8's.

Please let me know if you have any questions.