Time and Sales Issue IB Data Feed

[jbburchED]

Hello,

I have been noticing some huge prints in the time and sales window for the ES over the last week or so. They caught my eye because they were larger than the entire book on the DOM but the market hardly flinched. I have seen anywhere from 83K - 6K prints several times a day. They don't appear to be real given the market depth and the fact that the market hardly upticked/downticked (sometimes no change at all and there were only 800-1400 orders at each price level on the DOM).

I am connected to IB data feed. I know the data is filtered, so lets not go there quite yet....

I decided to bring up the TWS (IB) time and sales window and watch it next to the Ninjatrader times and sales window. I DONT see the erroneous prints in the IB window. Only the Ninjatrader window. I am a developer so I know how the NT window works. I know it is reading the OnMarketData method call to show the last market data type messages. However, I have also written entire applications that connect directly to the Interactive Broker's API directly. Given that the window in IB doesn't show the prints, and most all the data shown in TWS comes through the API, I am wondering if you have a bug in your IB connector.

Can you tell me which API version you have written the IB connector to? Also, can you tell me which TickType you are subscribing to in IB's API when you register using the reqMktData() method in the IB API's interface? Your developers will know.

Something isn't right. I can verify it by writing my own adapter and checking the values I am getting through the API vs. what NT is showing but I would rather not do that if I don't have to.

Regards,

Brian Burch

[jbburchED]

Just caught an example

See screen shot attached. This shows the NT window next to the IB window. Notice the large 5800 print on the NT window and no corresponding print on the IB window. Furthermore, if you review activity during the time, the market doesn't appear to be reacting to such large volume trades. There wasn't enough on the order book to absorb all of that without reacting.

I am sure if I were to download market replay data from kinetic, it would show that this print wasn't real. Thus, it would be easy to say that IB's data is bad. However, IB's own T&S window isn't showing it either.

Prints like this cause any volume related or order flow indicator to be completely bogus.

Thanks.

[NinjaTrader_Matthew]

Hi Brian,

The API version we use is 9.69. I'm not certain of the TickType at this moment, but we have researched this in the past and I'd be happy to provide you with the information we have.

Our experience shows that there are last trades reported in the TWS market data that are not shown in the TWS time and sales. (trades reported late, or spread trades or other undisclosed conditions)

TWS's T&S window runs on a different, less filtered stream than the Level 1 data stream NinjaTrader would access via the API, so there can be slight differences expected, which according to our observations can be more pronounced during rollover periods (ES 09-13 --> ES 12-13).

From our testing and comparisons, the NinjaTrader TWS API stream access matches what other 3rd party programs receive and display. This is dicussed in detailed in the following thread:

http://www.ninjatrader.com/support/forum/showthread.php?t=25700&page=6

If you would like clarification how the TWS feed used internally works, please follow up with IB support and they should be able to offer clarification.

Direct access to TWS's alternative feed is on our feedback list tracked using #1822, however I do not have an ETA on when this will be available. We will be retesting these scenarios in the next major release of NinjaTrader

Please let me know if you have any questions.

[jbburchED]

API Version?

Your response says that you use API version 9.69. However, IB's API release notes say that the date of that release is September 9th, 2013. I don't think Ninjatrader has release a software release since then?

The previous API version is 9.68 with a release date of October 2012.

Please clarify.

Regards.

[jbburchED]

Additional Clarification

Is NT trying to use a reconstruction of the tickSize() and tickPrice() asynchronous responses to build the time and sales or is NT requesting tick type 233 and processing the realtime volume tick type.

See the following from the IB API reference guide:

"You will receive the RTVolume update through the tickString() event within field value 48."

I suspect that you are doing the former and not the latter. The Time and Sales window seems to be using the tickString() field to show its data. By the way that data actually timestamps the last trade time based upon the API guide.

Anyway, I think that is probably the difference. The new method is much easier to deal with and appears to not have those erroneous prints in it.

Maybe this is what you mean by saying you have a feature request to implement the "less filtered" approach?

I don't think it really is a question of filtering. It is a question of trying to reconstruct the tape from a bunch of asynchronous replies that don't show every trade.

Regards.

[NinjaTrader_Matthew]

Sorry for my misunderstanding, the TWS socket server version is 67. We are not on the latest 69.

I'm not aware of the underlying details. The way it was described to me I interpreted as filtering; however call it what you will, it is expected as there is a different data feed for TWS than we used through NinjaTrader. What is added on our feedback list is giving users the ability to use the same source that TWS uses.

[cjvernon]

Any follow-ups?

I'm interested in this issue.

I build an algo client (C++) connected to the IB Gateway.

I use NT as my visual aid intraday since my client does not have graphical output.

In my client, I first tried first-differencing every second the (total) Volume ticktype from tickSize, to arrive at the volume traded over each second. I observed erratic such data, including many lulls of zeroes followed by a huge print. While there could, conceivably, be a thread syncing issue in my C++ code, I've heard others state the same anomalies.

So, I've now switched to using the RTVolume ticktype via tickString.

My data now matches the RT charts in TWS.

I would appreciate an answer as to where NT gets its 1-second volume data.

PS By the way, as an aside, the RTVolume data feed via tickString is bizarre, in that if you disconnect and reconnect it first sends you an aggregated catch-up volume. It is not a simple, mindless streaming service. I don't like this "feature". I don't want it thinking at all because that overhead impedes thruput and efficiency.

[NinjaTrader_Bertrand]

cjvernon, we would accumulate the realtime tick volume seen from IB's API based on time - the data in this setup would be locally timestamped by your PC clock.

[cjvernon]

Great- that is the way I do it too. Thanks.

[cjvernon]

I no longer believe this is what NT is doing.

To arrive at volume traded, I used to use the Total Volume feed (arrives in the "tickSize" call for those who know the IB API) and I would difference this value each second to arrive at the estimate sales for each second. When I did this, I would frequently see large spikes (every 3-5 seconds or so) not in any other data feed, spikes I attributed to "catchups" following heavy trading. I can't tell if the chart in NT has "erroneous" prints or prints that make up for prior prints that are "too small." I think it ls the latter- these prints "catch up" the (total) volumes.

I have an every 1-second chart running in NT7 in CME Crude Oil, and the Volume indicators for this chart is showing frequent spikes very similar to those I saw when I used to use the "Total Volume" feed and first-difference it. The spikes in my chart on NT are also displaced from the action, a second or more after the big moves. These spikes are not present in my trading client's reading of the feed from IB that arrives in "tickString" namely IB's more quasi-realtime feed.