Testing strategy on market replay gives worse fills on faster replay speeds

[Chrundle]

Hello,

I'm testing a strategy on Market Replay using Market Replay data that was downloaded from NT.

I am getting different fills during a live replay depending on the replay speed that was set.

What I did to verify the issue.

Started the Market Replay from the beginning on MAX speed.
Once the first order was filled and completed I stopped the replay. It entered long @ 3369.00 with a market order.

This didn't seem right since it should have been 1 tick above the previous bar's high.


I then restarted the Market Replay from the beginning (same starting spot as when testing it on max speed)
I ran the replay on high speeds until it got within a few minutes of the previous trade.

I then turned the speed down to "1x" and it entered long @ 3368.00 with a market order, in this case I got a whole point better on "1x" than when on "max"
I've tested this scenario multiple times and the same thing happens.


I noticed the issue due to awful fills when running the replay on high speeds.

I don't see how this could be related to my strategy since it's the same exact scenario, only with different speeds. This seems to be a bug with the market replay feature.

Thanks!

[NinjaTrader_ChrisL]

Hello Chrundle, thanks for your post.

The Replay connection will replay every tick in the market regardless of the speed that it's set to. Are you seeing ticks come in on a slower speed that are not coming in on max speed? Print out the condition set you are using and the time the condition set becomes true e.g.

Print("Checking Condition" + " " + Time[0]);
if(<condition1> && <condition2>)
{
Print("Condition True " + <condition1> + " " + <condition2> + " " + Time[0]);
EnterLong();
}

What differences do you see in max/minimum speed?

Are you able to see the same thing when testing the SampleMACrossover strategy?

I look forward to assisting.

[Chrundle]

Thanks for the quick response!

The ticks are coming out the same regardless of speed, the difference is when the order is being filled and the price at which it's filled at.
I set a variable to track the currentTime in the onBarUpdate and logged that time in the onOrderUpdate, when the order was filled.

Also, something to note. I thought it may have been a market replay issue (it's wonky sometimes when coming back to a day that was previously downloaded)
So I disconnected and connected to CQG to redownload the data for that day, but since it's just over 3 months, it wasn't available. I connected back to market replay with the same setup as before, and when on max speed the fill was at 3368.5 when before and in the screenshot above, it filled @ 3369

When at max speed, this was the results.

Checking Condition 10/5/2020 9:50:00 AM
Checking Condition 10/5/2020 9:50:00 AM
Checking Condition 10/5/2020 9:50:00 AM
Checking LONG Condition 10/5/2020 9:50:00 AM 3367.75
Checking Condition 10/5/2020 9:50:00 AM
Checking LONG Condition 10/5/2020 9:50:00 AM 3368
LONG Condition True 10/5/2020 9:50:00 AM 3368
Checking Condition 10/5/2020 9:50:00 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:02 AM
Checking Condition 10/5/2020 9:50:03 AM
Checking Condition 10/5/2020 9:50:04 AM
Checking Condition 10/5/2020 9:50:04 AM
Checking Condition 10/5/2020 9:50:04 AM
Checking Condition 10/5/2020 9:50:04 AM
Checking Condition 10/5/2020 9:50:04 AM
Checking Condition 10/5/2020 9:50:05 AM
Checking Condition 10/5/2020 9:50:06 AM
Checking Condition 10/5/2020 9:50:06 AM
Checking Condition 10/5/2020 9:50:06 AM
Checking Condition 10/5/2020 9:50:06 AM
Checking Condition 10/5/2020 9:50:06 AM
Checking Condition 10/5/2020 9:50:06 AM
Checking Condition 10/5/2020 9:50:06 AM
Checking Condition 10/5/2020 9:50:06 AM
Checking Condition 10/5/2020 9:50:07 AM
Checking Condition 10/5/2020 9:50:08 AM
Checking Condition 10/5/2020 9:50:08 AM
Checking Condition 10/5/2020 9:50:09 AM
Checking Condition 10/5/2020 9:50:09 AM
Checking Condition 10/5/2020 9:50:09 AM
Checking Condition 10/5/2020 9:50:10 AM
Checking Condition 10/5/2020 9:50:10 AM
Checking Condition 10/5/2020 9:50:10 AM
Checking Condition 10/5/2020 9:50:10 AM
Checking Condition 10/5/2020 9:50:10 AM
Checking Condition 10/5/2020 9:50:10 AM
Scalp order filled @ 3368.5 10/5/2020 9:50:10 AM - CurrentPrice 3368.5

When setting the speed to 1x before the order occurred...

Checking Condition 10/5/2020 9:50:00 AM
Checking Condition 10/5/2020 9:50:00 AM
Checking Condition 10/5/2020 9:50:00 AM
Checking LONG Condition 10/5/2020 9:50:00 AM 3367.75
Checking Condition 10/5/2020 9:50:00 AM
Checking LONG Condition 10/5/2020 9:50:00 AM 3368
LONG Condition True 10/5/2020 9:50:00 AM 3368
Checking Condition 10/5/2020 9:50:00 AM
Scalp order filled - 3368 10/5/2020 9:50:00 AM 3367.75
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:01 AM
Checking Condition 10/5/2020 9:50:02 AM

You can see the scalp order isn't filled for 10 seconds when on max speed, where it's instant while on 1x.

I also was not able to reproduce with the SampleMACrossover strategy.

[Chrundle]

I've been doing a bit more testing with the SampleMACrossover strategy and I'm definitely getting some weirdness.
I downloaded a fresh week of market replay data for early November. I ran the SampleMACrossover strategy on max and opened a window of the trade results after it got through about half of the day.
I then reset the playback to the start and recorded again on max and I'm getting different results every single time. Sometimes it's off by a second or two, sometimes more.
Also, when setting the replay back to the start of the day sometimes a very large bullish bar appears as it goes from historical to live replay, that wasn't there before
I let the replay play for a bit and reset it again to the start and it's gone again. This has happened multiple times now.

I tried redownloading the same week of replay data and still have the same exact issues.

Update: I cleared out the cache and redownloaded replay data again and still getting very different results each time I run the strategy.

[NinjaTrader_ChrisL]

Hello Chrundle, thanks for your reply.

Please make sure you are on the latest version of NinjaTrader 8. There were issues with the playback connection prior to 23.2. Please close out of NinjaTrader and install from the latest version below:

https://ninjatrader.com/GetStarted

Please test the playback connection again after updating to the latest version.

[Chrundle]

Oof, looks like that was the problem... I've been looking at the releases page for updates, but didn't realize it only lists the major versions and you need to drill down to see minor updates, oops.
Thanks for your help, Chris!