Why OnBarUpdate's and OnMarketData's data differ?

[quicktrick]

Hello!

Please, try the attached code with the settings shown on the picture.

Why on historical data OnBarUpdate's and OnMarketData's data differ so much?

OnBarUpdate threadId:    10
2017-04-13 09:49:58.5620000 (Unspecified) 2334,5 Historical
OnMarketData threadId:    10
2017-04-13 09:48:41.5970000 (Unspecified) 2334,5 Historical
OnBarUpdate threadId:    10
2017-04-13 09:49:58.5620000 (Unspecified) 2334,75 Historical
OnMarketData threadId:    10
2017-04-13 09:48:55.4490000 (Unspecified) 2334,75 Historical

Edit:
If you set 'Tools->Options->General->Time zone' to UTC, you will see even more interesting things around move from Historical to Realtime (try Realtime too).

[NinjaTrader_JessicaP]

Hello quicktrick, and thank you for your question.

I noticed you are using 1 range bars. Please, when testing incoming tick data, use one tick bars for this kind of testing.

This said, OnMarketData events occur on every piece of market data information given to you by your data feed provider. OnBarUpdate, meanwhile, contains information that has been interpreted by NinjaTrader and consolidated into a bar. This situation is analagous to comparing a news magazine article to breaking news on the T.V. The news magazine article is going to contain things like cross referencing things that happened for accuracy and related information for context, while the breaking news story is simply going to present things as they are happening leaving the interpretation to you. In this analogy, OnBarUpdate is similar to a magazine article and OnMarketData to the breaking news story. Interpreted open, high, low, and close data is going to be different from the composite ask and bid data as it comes in that comprises a bar.

If you can let us know, after testing with one tick data, any specific items we can look at, we will be happy to answer any questions we may in greater depth.

[quicktrick]

Jessica, it seems to me you want to get rid of me and uncomfortable questions. I don't understand why I have to use one tick bars. Range bars are totally suitable for that purpose. And please, try my code, you'll see the problems with time data yourself. Especially with its DateTime.Kind property.

You know... I write this for you, for your product, so you can fix its bugs. As for me, I can live with the bugs I mentioned here.

[NinjaTrader_JessicaP]

Your report is definitely appreciated, as are your efforts to improve NinjaTrader. We have always been very open to community feedback such as the feedback you are providing here, and have a history of acting on it when we are able to see the same thing on our end that you are seeing on yours.

My mentioning tick bars was an attempt to create a simple set-up that we could both use on our ends to make sure we are running the same tool the same way. When we are comparing OnBarUpdate to MarketReplay, if we set up a 1 tick bar, we can create situations where updates are more predictable and evenly spaced. If you can tell us the reason range bars are or should be used this will help us understand better what we should be reviewing. If you prefer using range bars, this was only a suggestion to make sure our test environments matched.

Can you help us understand what documented behavior it is we should be looking for? Our threading model is not documented, and can not be guaranteed. This said I realize that calling threading methods can reveal something else about the platform.

Once again, I deeply appreciate your efforts to improve the platform. It is just very difficult for us to make use of the information you provided without a clear case.

[quicktrick]

Jessica, I checked threads just to ensure, that OnMarketData events were fired sequentially and they couldn't overlap each other if data comes faster than it gets processed. I use tick data for my calculations and I use 1 Range chart to display their results because it seems impossible to get any viewable picture when using 1 Tick chart. I'm quite satisfied with 1 Range chart though it doesn't show me every single tick.

Another question I can't explain is why OnBarUpdate shows more ticks than OnMarketData in my test when there is quite a lot of historical data (e.g. 7 or more days of tick data). I guess you generate fake 1 Range bars to fill gaps if any. I haven't checked that because it's not interesting for me already, I'll just use OnMarketData.

[NinjaTrader_JessicaP]

Thank you for your most recent reply, I believe the following quote gives us some context as far as what you would like us to look at on our end.

Another question I can't explain is why OnBarUpdate shows more ticks than OnMarketData in my test when there is quite a lot of historical data (e.g. 7 or more days of tick data). I guess you generate fake 1 Range bars to fill gaps if any. I haven't checked that because it's not interesting for me already, I'll just use OnMarketData.

If we review the documentation for Multi-Timeframe and Instruments (aware of the fact that we are using a single time frame indicator), we find

Data processing sequence

Understanding the sequence in which bars series process and the granularity provided by market data vendors is essential for efficient multi-series development. Let’s assume we have two series (primary and secondary) in our script, which is representing the same instrument, yet different intervals. During historical data processing, NinjaTrader updates the two series strictly according to their timestamps, calling the primary bar series of the corresponding timestamps first, and then calling the secondary series.

Note: Historical bars are processed according to their timestamps with the primary bars first, followed by the secondary, which is NOT guaranteed to be the same sequence these events occurred in real-time. If your development requires ticks to process in the same sequence historically as well as in real-time, you will need to enable Tick Replay (utilizes more PC resources).

This diagram from the same page shows why you would see a different number of ticks compared to market data events; just treat the other series as your Market Data events instead, but pay attention to the time sequencing. Multiple market data events occur within the same millisecond, and therefore have the same time sequence.



When you examine this on your own please use DateTime.ToString("dd MMM yyyy HH:mm:ss.ffff") as this will print timestamps down to tenth of a millisecond accuracy.

Please let us know what we can clarify or investigate further.

[quicktrick]

Thank you, Jessica! I know the things you mentioned. The question was why we have more OnBarUpdate price change events than OnMarketData price change events. I can explain it only that NT generates fake 1 Range bars to fill the gaps. But I haven't investigated that, because it's not so interesting to me as I don't use OnBarUpdate events at all.