Discrepancy between historical data and real time data in IB

[nikolaalx]

Hi everyone,
I have been spending a lot of time doing research and optimizing my strategy.

I however noticed that there is a discrepancy between the historical data and real time data. I use IB for both real time and historical data.

The discrepancy is not major, but is sufficient to cause some deviation from the trades that are reported in my optimization summary.

This gets me real worried to the overall credibility of all the data and work that I have put so far...keeping in mind that the data that I have been working on is well...compromised in a certain way.

Is there any advice that someone can give me? Has anyone else noticed this?

I am thinking of switching my historic data to Kinetick, however i am again worried that it could have a different data than IB where I am executing my trades in real time.

[NinjaTrader_Bertrand]

nikolaalx, you generally want to keep those items in mind we have compiled here - https://www.ninjatrader.com/support/...ime_vs_bac.htm

IB's data is also snapshot data and locally timestamped for your received realtime data.

A higher performing 'true' datafeed like Kinetick would provide the data you work with unfiltered and natively timestamped.

[nikolaalx]

I have purchased a subscription from Kinetick, and the problem still remains.

In fact I find the historical data from Kinetick identical to the one of IB.

[NinjaTrader_Bertrand]

In which order do you now connect your feeds then?

[nikolaalx]

I disconnect from IB and leave only Kinetick as sole connection.

I test/optiimize and after that connect to market replay.

In the end I compare the two.

I am attaching a screenshot where you can see how significant the deviation is.

[NinjaTrader_Bertrand]

nikolaalx, I think that would be more of an issue with how much historical you load up before actually running the session in replay mode then. For the replay data itself, did you record that running live on Kinetick?

[nikolaalx]

The replay data was recorded from IB real trading (I save every day I trade).

No matter how many days I load the data stays like this.

The historical data from Kinetick shows the exact same deviation as the historical data from IB.

I have flushed/reloaded all historical data once I purchased the kinetick subscription.

[NinjaTrader_Bertrand]

Generally I would not expect the data between providers to match up exactly nor would I expect that from the results coming from replay (that would have the intrabar formation) vs backtested results. The areas with the aroon value deviations though seem to be confined to a period after the replay session open / gap - would you see the identical gap reviewing your Kinetick backtesting chart? Also - is the session template set the same as in backtesting?

[brynprice]

Nikolaalx, did you solve this issue in the end? I am going through a similar panic right now. All of my backtesting was done using IB's historical data. When I try to trade using their real-time data, I will often find myself either missing my entries or entering a trade when I shouldn't have. When I "refresh" the chart it often looks totally different and it might well be the case that according to the refreshed data, price didn't even hit my entry price after all. Now that I am in the process of automating my strategy, I am trying to decide whether to include instructions to make historical data requests every few seconds in my code (in order to mimic refreshing the screen when I trade manually), or to purchase a completely different data feed entirely and use that.

[koganam]

Nikolaalx, did you solve this issue in the end? I am going through a similar panic right now. All of my backtesting was done using IB's historical data. When I try to trade using their real-time data, I will often find myself either missing my entries or entering a trade when I shouldn't have. When I "refresh" the chart it often looks totally different and it might well be the case that according to the refreshed data, price didn't even hit my entry price after all. Now that I am in the process of automating my strategy, I am trying to decide whether to include instructions to make historical data requests every few seconds in my code (in order to mimic refreshing the screen when I trade manually), or to purchase a completely different data feed entirely and use that.

For live intraday trading, you will almost always be better off simply getting an unfiltered data feed. That is a feed that is certainly not IB.

[brynprice]

Thank you Koganam for replying to both my posts. I might very well end up using an unfiltered feed like Kinetick but my plan at the moment is to have my program fire of historic data requests at regular intervals throughout the day so as to ensure that the data is constantly being converted from filtered real time packets to the historical data format (however it may be that they produce it) that I have always used. Any time an entry signal appears to have presented itself, I'll need to make one final historical request to confirm that it is indeed an entry according to the "historical data" (historical even though it has just occurred a second ago) - and if I get confirmation then in I get. This is pretty ugly, isn't it and I might be a second or too "late" if I try to do it this way. But as long as it is only a second or two I should be ok. I do have my reasons for wanting to try this method first, namely:

- I spend most of my time in Europe/Asia far away from the US servers and I suspect my computer might not be able to handle the unfiltered data (perhaps this concern is unfounded - haven't tried it yet so I have no idea);
- If at all possible I would rather use the exact same data as that which I have always used for my backtesting (IB historical data); this is not absolutely crucial to my success and the system should still "work" but I would have to make a lot of changes nonetheless as I don't want to leave the slightest thing to chance;
- the cost of the datafeed is also a factor until I start to be profitable. If and when that happens, then cost will presumably cease to be a factor.