duplicate trades

[greentrader]

comparison with sierrachart

For NQ, have so far seen 2 missing trades and 1 duplicate trade in overnight trading. SierraChart capturing the same tick data on the same machine with the same ISP had no missing trades and no duplicate trades when compared to the IQFeed's historical data. So the problem is very likely with NT's OnMarketData messaging.

[NinjaTrader_JoshP]

greentrader,

Let me try and reiterate. You cannot compare across other platforms. That is not an apples to apple comparison. Furthermore, you cannot compare historical data against incoming real-time data. It is very possible they pull from different data servers and many data providers are known to have slightly different data across their servers.

[greentrader]

on the contrary

greentrader,

Let me try and reiterate. You cannot compare across other platforms. That is not an apples to apple comparison. Furthermore, you cannot compare historical data against incoming real-time data. It is very possible they pull from different data servers and many data providers are known to have slightly different data across their servers.

There is not a shred of evidence to suggest that NT receives inferior realtime data (with duplicate trades) when compared to SierraChart.

On the contrary, the evidence in this thread strongly suggests that NT's OnMarketData messaging is flawed, occassionally producing bogus trade data. The exact nature of this flaw has yet to be determined, and may or may not be tied to the way I'm using NT.

I've got some more experiments in mind for tonight. But, honestly, it should be straightforward at this point for anyone with the source to NT to set up a system that looks for a discrepancy between incoming trade data and the messages generated by OnMarketData (run it for at least 24 hours, using index futures). In fact, I would have thought such a system would already have been created somewhere to verify all the messaging done by NT (in which case I could just give you a strategy to be tested without having to include all the file handling and comparisons I'm now stuck with, since I don't have direct access to the incoming trade data).

[NinjaTrader_JoshP]

greentrader,

NinjaTrader just acts as a radio receiver. Whatever is received is displayed. There is no manipulation. What you get from your data provider is what you see.

If you still feel there is an issue please provide a reference sample demonstrating what you feel is a duplicate tick.

[greentrader]

not a radio receiver

NinjaTrader just acts as a radio receiver. Whatever is received is displayed. There is no manipulation. What you get from your data provider is what you see.

The incoming data is queued and distributed to callbacks like OnMarketData. Plenty of room for mistakes to be made, and is far from being a radio receiver.

Without seeing NT's incoming data, it's impossible for me to absolutely prove that the duplicate trades are not from the feed, especially if you're unwilling to consider comparable feed to other apps from IQFeed on the same machine at the same time using the same ISP!

I'm going to (eventually) figure this out, or find a workaround. Will let you know what I learn from next set of experiments.

[NinjaTrader_JoshP]

greentrader,

By no means are we unwilling to look into it. We need reproducible scenarios demonstrating what you feel is wrong and then we can evaluate it. Thank you.

[greentrader]

latest evidence

I put together a simpler strategy with 3 instruments (ES, NQ, YM), which did nothing except output trade data from OnMarketData (OMD) to text files that could be later compared to historical IQFeed data. After two days manipulating various settings, my tentative conclusion is that the bug is related to use of the CalculateOnBarClose (COBC) setting.

With COBC = false, no duplcate trades are seen.

With COBC = true, duplicate trades are occassionally sent to OMD, as illustrated by the following real and duplicate trade pairs seen in the recent overnight session:

for ES (system time, exchange time, last price, volume):

4/16/2009 7:32:38 PM, 4/16/2009 7:32:37 PM, 859, 4
4/16/2009 7:32:52 PM, 4/16/2009 7:32:37 PM, 859, 4

4/17/2009 3:47:47 AM, 4/17/2009 3:47:46 AM, 855.75, 2
4/17/2009 3:47:52 AM, 4/17/2009 3:47:46 AM, 855.75, 2

for NQ:

4/16/2009 7:33:00 PM, 4/16/2009 7:32:59 PM, 1345, 1
4/16/2009 7:33:12 PM, 4/16/2009 7:32:59 PM, 1345, 1

for YM:

4/16/2009 7:37:10 PM, 4/16/2009 7:37:09 PM, 8046, 1
4/16/2009 7:44:28 PM, 4/16/2009 7:37:09 PM, 8046, 1

where the duplicates are characterized by the fact that the time they are seen by OMD is many seconds later than their exchange time.

So I'm currently running my more complex trading strategy with COBC = false, and, so far, have not seen any duplicate trades. If, after the next couple of days, I see no further duplicates, then I'll be satisfied that setting COBC = false is a satisfactory workaround to the problem.

The original idea behind setting COBC to true was to prevent a lot of unnecessary calls to OnBarUpdate, which I wasn't using. But apparently this setting triggers a bug in NT's queue management, resulting in duplicate trades (and perhaps other garbage) being sent to OMD.

[NinjaTrader_JoshP]

greentrader,

What you should do to compare is not your output files to historical data since historical data servers are usually different than real-time servers and can give different data sets. Instead you will want to just compare your COBC = false against your COBC = true output. Run both at the same time.

[greentrader]

hmmmm

What you should do to compare is not your output files to historical data since historical data servers are usually different than real-time servers and can give different data sets. Instead you will want to just compare your COBC = false against your COBC = true output. Run both at the same time.

Well, sure, there are lots of little (24 hour) experiments I could do to help you guys further sort this out, but THAT'S NOT MY JOB. And to suggest at this point that the bug could still be due to bad data from IQFeed is ridiculous. DTN/IQFeed denies sending duplicate trades, other apps don't see such trades, NT (so far) doesn't see them with COBC = false, etc.

Historical data servers may differ from realtime feed, but (based on my detailed tracking) rarely so for the index futures (from IQFeed), and not by producing delayed, duplicate orders like those reported in this thread.

The nicest thing I have to say at this point is that this thread is an exception to the otherwise great job you guys usually do providing support for NT.

[NinjaTrader_JoshP]

greentrader,

What you have been doing the whole time have been apples to oranges comparisons. From our experiences and that of our other customers there have been no other reports coinciding with yours. As such all we have been trying to do this whole time was get you to reach a baseline scenario.

We have also requested you for your test scenarios several times, but you have never posted them and as such we cannot reproduce your scenario.

[solvmatic]

I found that only when adding multi-time in OnBarUpdate() will get get duplicate data from OnMarketData. I guess it might be because OnMarketData info has to feed the multi-time processes in OnBarUpdate().

[NinjaTrader_JoshP]

solvmatic,

Correct, this can be the case for multi-time indicators.