Market replay events not raised in correct sequence

[ScoobyStoo]

Hi NT guys,

Just running some market replay tests based on data obtained from a ZenFire feed and running into some sequencing problems.

Please take a look at the attached event sequence dump. Looks suspiciously like the market replay engine is raising all the MarketData events for a particular one second interval and then raising all the MarketDepth events for the same interval. This obviously is significantly different from the order in which the events were originally raised, with the MarketData and MarketDepth tightly synchronised (apart from the odd instance where the multithreading causes a bit of lead/lag).

Can you confirm that this is the case?

[NinjaTrader_Bertrand]

ScoobyStoo, could you please repost the screenshot?

[ScoobyStoo]

ScoobyStoo, could you please repost the screenshot?

Sorry Bertrand, I actually forgot to attach it. Been a very long day...

Here it is.

[NinjaTrader_Bertrand]

Thanks - on which chart interval did you collect this? Which code did you use to 'Print' your time for debugging?

[ScoobyStoo]

Thanks - on which chart interval did you collect this? Which code did you use to 'Print' your time for debugging?

I've attached 2 files. One containing the custom indicator code used to dump the event sequence, and one containing a cleaned up output. They have been generated from a 100 tick chart of the 6E contract on 09/07/2009 at 9.30am GMT+1.

As you can see, there are issues with the timestamping. The MarketData events only appear to be timestamped to an accuracy of 1 second and the MarketDepth events appear in blocks which all have the same timestamp.

If I had to guess I'd say that the events aren't being stored in the database correctly (they should be stored using ticks as this gives the maximum granularity permitted by the DateTime structure). Then when the market replay engine pulled them out of the database they'd be fired in the correct order.

Although having said this I often see on the ZenFire feed at very active times that multiple events (usually MarketDepth) are raised within the 1 tick (100 nanosecond) timescale. Obviously these events are a bit more tricky to handle and reproduce correctly...

[NinjaTrader_JoshP]

Accuracy goes to the seconds only here. Replays do not pull from the database, they pull from the replay file which contains information of the exact order in which ticks are received and just plays it back as is.

[ScoobyStoo]

Accuracy goes to the seconds only here. Replays do not pull from the database, they pull from the replay file which contains information of the exact order in which ticks are received and just plays it back as is.

Thanks for the quick reply Josh.

However, the don't see the market replay event sequence being the same as the live event sequence. To verify this for yourself please look at the attached files which cover a 40 second interval of the 6E contract this morning.

Live = Live data received from a ZenFire feed
Replay = Same data replayed through market replay

Notice that in the live data the event sequence is such that the MarketDepth (Level II) event is raised first and then the MarketData (Level 1) event is raised immediately afterwards...

OnMarketDepth()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Ask
Market Data Operation : Update
Market Data Position : 0
Market Data Price : 1.4219
Market Data Volume : 25


OnMarketData()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Ask
Market Data Price : 1.4219
Market Data Volume : 25


OnMarketDepth()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Ask
Market Data Operation : Update
Market Data Position : 0
Market Data Price : 1.4219
Market Data Volume : 26


OnMarketData()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Ask
Market Data Price : 1.4219
Market Data Volume : 26


OnMarketDepth()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Ask
Market Data Operation : Update
Market Data Position : 2
Market Data Price : 1.4221
Market Data Volume : 56


OnMarketDepth()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Ask
Market Data Operation : Update
Market Data Position : 3
Market Data Price : 1.4222
Market Data Volume : 45


OnMarketDepth()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Bid
Market Data Operation : Update
Market Data Position : 3
Market Data Price : 1.4215
Market Data Volume : 63


OnMarketDepth()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Bid
Market Data Operation : Update
Market Data Position : 0
Market Data Price : 1.4218


OnMarketData()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Bid
Market Data Price : 1.4218
Market Data Volume : 2


Contrast this with the exact same data replayed through the market replay engine...

OnMarketData()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Ask
Market Data Price : 1.4219
Market Data Volume : 25


OnMarketData()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Ask
Market Data Price : 1.4219
Market Data Volume : 26


OnMarketData()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Bid
Market Data Price : 1.4218
Market Data Volume : 2


OnMarketData()
Market Data Time : 09:17:37.0
Market Data Time (ticks) : 633837646570000000
Market Data Type : Bid
Market Data Price : 1.4218
Market Data Volume : 3


OnMarketDepth()
Market Data Time : 09:17:36.771
Market Data Time (ticks) : 633837646567717500
Market Data Type : Ask
Market Data Operation : Update
Market Data Position : 0
Market Data Price : 1.4219
Market Data Volume : 25


OnMarketDepth()
Market Data Time : 09:17:36.771
Market Data Time (ticks) : 633837646567717500
Market Data Type : Ask
Market Data Operation : Update
Market Data Position : 0
Market Data Price : 1.4219
Market Data Volume : 26


OnMarketDepth()
Market Data Time : 09:17:36.771
Market Data Time (ticks) : 633837646567717500
Market Data Type : Ask
Market Data Operation : Update
Market Data Position : 2
Market Data Price : 1.4221
Market Data Volume : 56


OnMarketDepth()
Market Data Time : 09:17:36.771
Market Data Time (ticks) : 633837646567717500
Market Data Type : Ask
Market Data Operation : Update
Market Data Position : 3
Market Data Price : 1.4222
Market Data Volume : 45


OnMarketDepth()
Market Data Time : 09:17:36.771
Market Data Time (ticks) : 633837646567717500
Market Data Type : Bid
Market Data Operation : Update
Market Data Position : 3
Market Data Price : 1.4215
Market Data Volume : 63


OnMarketDepth()
Market Data Time : 09:17:36.771
Market Data Time (ticks) : 633837646567717500
Market Data Type : Bid
Market Data Operation : Update
Market Data Position : 0
Market Data Price : 1.4218
Market Data Volume : 2


You can see that the replay data starts with 4 MarketData events and then the MarketDepth events are playing catch-up afterwards. They are completely out of sequence.

[ScoobyStoo]

And if you were wondering why this is a serious problem then take a look at the attached sample market replay event sequence, generated using the updated attached indicator.

You can see that all the MarketData events are raised for the 1 second interval and only then are all the MarketDepth events raised. This means that during market replay it is next to impossible to reconcile L1 data with L2 data because they are totally out of synch.

These leads to instances such as...

OnMarketDepth()
Current Ask Price : 1.3965
Current Bid Price : 1.3963
Market Data Time : 09:30:06.894
Market Data Time (ticks) : 633827286068946590
Market Data Type : Ask
Market Data Operation : Update
Market Data Position : 0
Market Data Price : 1.3964
Market Data Volume : 1

...where the ask/bid price in the L1 data (obtained by calling GetCurrentAsk() and GetCurrentBid()) was updated to 1.3965/1.3963 long ago, but the L2 data still has the prevailing ask price as 1.3964.

Can I please make 2 requests for NT7 (if it's not too late)

1. That all events are timestamped to the maximum granularity (i.e. using DateTime.Tick). 1 second granularity is very limiting when trying to do realtime reconciliation between the L1 and L2 datastreams.

2. The market replay functionality actually replays events in exactly the order they originally occured (thus keeping the L1 and L2 datastreams in synch).

I fully appreciate this won't be an issue for 99.9% of your customers but hopefully your developers will appreciate that fixing this up will really really help the 0.1% of your customers who are serious quants and engage in low-level programming using NT.

[NinjaTrader_Bertrand]

ScoobyStoo, thanks for posting the files and indicator, I'll test this on my end and get back to you.

[ScoobyStoo]

ScoobyStoo, thanks for posting the files and indicator, I'll test this on my end and get back to you.

Appreciate your help Bertrand.

Thanks

[NinjaTrader_Bertrand]

ScoobyStoo, I could not reproduce this on my end here with todays ES 09-09 tickdata, I get the same sequence of events whether I ran your test dump indicator on replay or live data with ZenFire. Which NinjaTrader and .NET version are you testing this on?

[ScoobyStoo]

ScoobyStoo, I could not reproduce this on my end here with todays ES 09-09 tickdata, I get the same sequence of events whether I ran your test dump indicator on replay or live data with ZenFire. Which NinjaTrader and .NET version are you testing this on?

Installed on the machine are:

NT v6.5.1000.9

.NET v1.0.3705
.NET v1.1.4322
.NET v2.0.50727
.NET v3.0
.NET v3.5

[NinjaTrader_Bertrand]

Then I would suggest to install the latest 6.5.1000.11 release using the Help > Download Site option in the Control Center, then recheck if the issue persists. If it does, it would be helpful to install the .NET framework fresh.

[ScoobyStoo]

Then I would suggest to install the latest 6.5.1000.11 release using the Help > Download Site option in the Control Center, then recheck if the issue persists. If it does, it would be helpful to install the .NET framework fresh.

Hi Bertrand,

Did a fresh WinXP build with the latest version of .NET v2.0 framework and NT v6.5.1000.11. Same issue still occurs with the ordering of the events.

Are you guys able to replicate it on any of your QA machines?

[NinjaTrader_Bertrand]

ScoobyStoo, so far have not been able to reproduce, but will also check on a XP machine here.