OnMarketData, ToString, seeing duplicates, multiple lines same timestamp

[balltrader]

HI,
In trying to understand the OnMarketData event, I started writing to the output window using:

protected override void OnMarketData(MarketDataEventArgs e)
		{   	string event_string = e.ToString();
			Print(DateTime.Now.ToString("HH:mm:ss.ffffff") + " OnMarketData event , Bar# " + CurrentBar + ", event time: " + e.Time + ", event type: " + e.MarketDataType + ", event string: " + event_string );

when real time data started appearing the output window, I noticed what appeared to be multiple lines of data that appears to be duplicates.

20:26:00.000000 OnBarUpdate event , Bar# 1318
20:26:00.000000 OnBarUpdate event , Bar# 1318
20:26:00.000000 OnBarUpdate event , Bar# 1318
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Last, event string: instrument='GC 12-17 Nymex' type=Last price=1278.6 volume=1 time='11/15/2017 8:21:41 PM' bid=1278.5 ask=1278.6 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Last, event string: instrument='GC 12-17 Nymex' type=Last price=1278.6 volume=1 time='11/15/2017 8:21:41 PM' bid=1278.5 ask=1278.6 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Last, event string: instrument='GC 12-17 Nymex' type=Last price=1278.6 volume=1 time='11/15/2017 8:21:41 PM' bid=1278.5 ask=1278.6 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Last, event string: instrument='GC 12-17 Nymex' type=Last price=1278.6 volume=1 time='11/15/2017 8:21:41 PM' bid=1278.5 ask=1278.6 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: DailyVolume, event string: instrument='GC 12-17 Nymex' type=DailyVolume price=0 volume=10536 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: DailyVolume, event string: instrument='GC 12-17 Nymex' type=DailyVolume price=0 volume=10536 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: DailyVolume, event string: instrument='GC 12-17 Nymex' type=DailyVolume price=0 volume=10536 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: DailyVolume, event string: instrument='GC 12-17 Nymex' type=DailyVolume price=0 volume=10536 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1278.7 volume=31 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1278.7 volume=31 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1278.7 volume=31 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.829010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1278.7 volume=31 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.830010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1278.5 volume=25 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.830010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1278.5 volume=25 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.830010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1278.5 volume=25 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20:21:33.830010 OnMarketData event , Bar# 1318, event time: 11/15/2017 8:21:41 PM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1278.5 volume=25 time='11/15/2017 8:21:41 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False

Data Provider is Continuum

Is this normal?
When trying to calculate tick volume at bid or ask, this appears to be wrecking my calculations by tripling trade transaction volume.

What do you suggest I do to start troubleshooting?
I have closed all other charts, so only 1 chart is open, script editor, control center, and output window.
I have had NT8 running continuously for 3 days, should I reboot for a fresh start?
Could these duplicates be caused by additional socket connections to the data provider being opened without the original socket connections being closed?

Does a typical marketdata stream contains such a firehose of data? wow

[NinjaTrader_AlanP]

Hello balltrader,

Thank you for your note.

OnMarketData is going to be called on every change in level one market data for the underlying instrument. For example all things remaining equal, if someone offers 1 contract 2 ticks off the best offer, OnMarketData will be called. If someone cancels a working offer in the book, OMD will be called.

I would suggest adding a filter before your code to prevent this being called so often, you could even require the current bar change before information is stored.

Please see the following link,

https://ninjatrader.com/support/helpGuides/nt8/en-us/?onmarketdata.htm

Please let us know if you need further assistance.

[balltrader]

yes i need further assistance.

I think I understand when OMD will be called.

Maybe I did not explain my question well enough.
==========================================
When this OMD event is triggered, I am reading the contents of the stream.

For each event, I display the timestamp using milliseconds

Within the same millisecond timestamp, I see multiple event types (Daily Volume, Bid, Ask)

For each event type, I see what appears to be duplicate data rows.

================================================== =======
In the latest capture, I see:

11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: DailyVolume, event string: instrument='GC 12-17 Nymex' type=DailyVolume price=0 volume=236160 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: DailyVolume, event string: instrument='GC 12-17 Nymex' type=DailyVolume price=0 volume=236160 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1288.7 volume=41 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1288.7 volume=41 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1288.8 volume=36 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1288.8 volume=36 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1288.8 volume=37 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1288.8 volume=37 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1288.8 volume=38 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:20:05.945564 OnMarketData event , Bar# 1236, event time: 11/17/2017 11:20:08 AM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1288.8 volume=38 time='11/17/2017 11:20:08 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False

==============================================

You can see the duplicate rows have the same time stamp, same event type, and same property values.

My understanding is: it appears no new market information was generated to create a new OMD event, since the values are the same, with the same millisecond timestamp.


Q 1: Is this an error, or is this how it will be?

Q 2: If this is how it will be, referencing the statement from your previous reply,

I would suggest adding a filter before your code to prevent this being called so often, you could even require the current bar change before information is stored.

do you mean I would need to use an IF() statement to determine if the timestamp of the row I am reading is different than the previous row I just read, and the event type is different, and the properties are all different, just so I can ensure my indicator and strategy calculations are not incorrect due to duplicate data?

[balltrader]

I just noticed the clock on my pc was showing a delay of 3 seconds from the OMD event timestamp.

I changed my Windows time server from time.windows.com to tock.usno.navy.mil

and now the newest data stream has event times that match closer to the computer's clock.

but the duplicate rows are still there.

11:40:10.403064 OnMarketData event , Bar# 1240, event time: 11/17/2017 11:40:10 AM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1288.9 volume=38 time='11/17/2017 11:40:10 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:40:10.403064 OnMarketData event , Bar# 1240, event time: 11/17/2017 11:40:10 AM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1288.9 volume=38 time='11/17/2017 11:40:10 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:40:10.403064 OnMarketData event , Bar# 1240, event time: 11/17/2017 11:40:10 AM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1289 volume=18 time='11/17/2017 11:40:10 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:40:10.403064 OnMarketData event , Bar# 1240, event time: 11/17/2017 11:40:10 AM, event type: Ask, event string: instrument='GC 12-17 Nymex' type=Ask price=1289 volume=18 time='11/17/2017 11:40:10 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:40:10.403064 OnMarketData event , Bar# 1240, event time: 11/17/2017 11:40:10 AM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1288.9 volume=37 time='11/17/2017 11:40:10 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
11:40:10.403064 OnMarketData event , Bar# 1240, event time: 11/17/2017 11:40:10 AM, event type: Bid, event string: instrument='GC 12-17 Nymex' type=Bid price=1288.9 volume=37 time='11/17/2017 11:40:10 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False

[NinjaTrader_Jim]

Hello balltrader,

I am responding to you on behalf of my colleague Alan who is out of the office today.

I am not seeing duplicates when I print out OnMarketData(). I tested by printing the time of the iteration down to the millisecond along with the data from the MarketDataUpdate event.

protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
{
	Print(marketDataUpdate.Time.ToString("yyyyMMddHHmmssfff") + " " + marketDataUpdate.ToString());
}

20171117135845041 instrument='ES 12-17 Globex' type=Ask price=2581 volume=352 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135845526 instrument='ES 12-17 Globex' type=Bid price=2580.75 volume=243 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171116190000000 instrument='ES 12-17 Globex' type=DailyHigh price=2586 volume=0 time='11/16/2017 7:00:00 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171116190000000 instrument='ES 12-17 Globex' type=DailyLow price=2577.75 volume=0 time='11/16/2017 7:00:00 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171116190000000 instrument='ES 12-17 Globex' type=DailyVolume price=0 volume=808718 time='11/16/2017 7:00:00 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135839646 instrument='ES 12-17 Globex' type=Last price=2580.75 volume=1 time='11/17/2017 1:58:39 PM' bid=2580.75 ask=2581 isReset=False
20171115190000000 instrument='ES 12-17 Globex' type=LastClose price=2585 volume=0 time='11/15/2017 7:00:00 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171116190000000 instrument='ES 12-17 Globex' type=Opening price=2584.25 volume=0 time='11/16/2017 7:00:00 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117091815876 instrument='ES 12-17 Globex' type=OpenInterest price=0 volume=3219914 time='11/17/2017 9:18:15 AM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171116191709892 instrument='ES 12-17 Globex' type=Settlement price=2585 volume=0 time='11/16/2017 7:17:09 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135845719 instrument='ES 12-17 Globex' type=Ask price=2581 volume=356 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135845735 instrument='ES 12-17 Globex' type=Bid price=2580.75 volume=240 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135845735 instrument='ES 12-17 Globex' type=Bid price=2580.75 volume=238 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135845736 instrument='ES 12-17 Globex' type=Bid price=2580.75 volume=228 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135845736 instrument='ES 12-17 Globex' type=Bid price=2580.75 volume=216 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135845736 instrument='ES 12-17 Globex' type=Bid price=2580.75 volume=204 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135845739 instrument='ES 12-17 Globex' type=Bid price=2580.75 volume=192 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135845784 instrument='ES 12-17 Globex' type=Bid price=2580.75 volume=191 time='11/17/2017 1:58:45 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135846551 instrument='ES 12-17 Globex' type=Ask price=2581 volume=357 time='11/17/2017 1:58:46 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False
20171117135847163 instrument='ES 12-17 Globex' type=Ask price=2581 volume=360 time='11/17/2017 1:58:47 PM' bid=-1.79769313486232E+308 ask=-1.79769313486232E+308 isReset=False

Could you perform the same test on your end and let me know if you are still seeing duplicates?

I look forward to being of further assistance.