Price data differs from that reflected in real time

[ラリー]

The prices on the chart are different from the prices reflected in real-time and the results of a manual reload.

I am supplying IB Securities market data to Ninja Trader, and the chart data in IB Securities TWS is the same as the result of reloading the chart on the Ninja Trader platform.

[NinjaTrader_Jason]

Hello ラリー,

Unfortunately the realtime data that NinjaTrader receives from the IB connection is filtered, while historical data is not. So when you reload historical data in the NinjaTrader chart, it then shows the same data as in TWS.

[ラリー]

Thank you for your reply.

Why is it filtered during real time?
Is there any way to prevent it from filtering or is there an alternative?

[NinjaTrader_Jason]

Can you please address the first question to Interactive Brokers.

You could connect to a different broker/data feed in the NinjaTrader platform that does not filter realtime data. Our preferred market data service is Kinetick. Market data services start as low as $69 per month. If you trade CME Globex products, you can have these exchange fees waived if you have a qualified brokerage account. This service is only available for NinjaTrader. Please visit www.kinetick.com for more detailed information.

[ラリー]

Thank you for your reply.

Understood.
I will ask IB Securities.

I will also ask Kinetick about the market data they provide.

[ラリー]

Thank you for your assistance.

I asked IB Securities why it is filtered during real time, etc. and received the following response.
I have also omitted the following due to the length of the exchange.

A: My understanding is that the JPX data is raw data from the exchanges, but is it filtered on the NinjaTrader side or on our side?

This is the answer.

I was not clear, but IB Securities provided us with the following link, which we found regarding filtering.

https://interactivebrokers.github.io...ical_data.html

I was also informed that Kinetic does not provide market data for Japanese stocks.


I would like to confirm, am I correct in assuming that IB Securities is filtering on their side?

Also, I found an article about filtering in the NinjaTrader documentation, can you help me with this (link below)?

https://ninjatrader.com/support/help...marketdata.htm

https://ninjatrader.com/support/helpGuides/nt8/real_time_tick_filter.htm?zoom_highlightsub=filter

[NinjaTrader_Jason]

The link handles historical data, while your question refers to realtime data. The NinjaTrader platform does not filter any data - it shows data as provided by the connected broker/data feed connection.

You can enable filtering of bad ticks on realtime data in the NinjaTrader platform using the 'Filter bad ticks' setting. Please see the link below for more information.
https://ninjatrader.com/support/help...ime_tick_filte r.htm

Enables or disable filtering of bad ticks. This filtering only works on real-time data and will filter ticks that are a greater then a set percentage away from the last tick. Set the percentage for filtering with the property: "% off market". Note: If NinjaTrader receives 2 or more ticks that violate the tick filter we will no longer filter the ticks as the market is assumed to have legitimately gapped up or down.

The '% off market' setting determines the offset percentage. For example, if you enter a value of 3, it will filter out a tick that is 3% or more away from the prior tick. If the prior tick was a price of 1000.00, it will exclude a tick of 970.00 or lower and 1030.00 or higher.

[ラリー]

Thank you for your reply.

The link handles historical data, while your question refers to realtime data. The NinjaTrader platform does not filter any data - it shows data as provided by the connected broker/data feed connection.

We will inform IB Securities that the Ninja Trader platform will not be filtered.

We will also attempt to set up real-time bad ticks and will contact you including communication with IB Securities.

[ラリー]

Thank you for your assistance.

We are contacting you to report the results of the filtering verification and IB Securities' response.


Thank you for your assistance.

We are contacting you to report the results of the filtering verification and IB Securities' response.

We have tried filtering 0.1, 0.3, 3, and 10% respectively, but the prices and graphs do not match when compared to the TWS chart, although it does not seem to be the case for all instruments.

Also, similar prices, chart graphs, etc. match when comparing TWS with the JPX's official Tokyo Stock Exchange website.


株価検索 | 日本取引所グループ (jpx.co.jp)

Below is the details of our communication with IB Securities.

We apologize for the inconvenience and thank you for your understanding.


IB Securities

We have read your inquiry, but we do not have a clear understanding of your question.
Ninja Trader's product is not ours, so we cannot tell you why Ninja Trader's chart is different from ours.
I think we also agreed that when we looked back at the previous charts, our TWS representation did not differ from the daily data listed on the Tokyo Stock Exchange (JPX).
I believe we also agreed that there was no problem when I checked our sample API on my PC and asked Ninja Trader to check it.
Can you please tell us what we need to check from the information you have provided?
Also, could you please use our sample API and see if it is similar to NinjaTrader's tool, as opposed to TWS?
If the display of our sample API and NinjaTrader's API are the same and different from TWS, we can ask the backend side to check if there is any bug in the display from TWS to the API.



I: I am

＞I also think we are in agreement that when we reviewed the previous chart, our TWS representation did not differ from the daily data as listed on the Tokyo Stock Exchange (JPX).

Yes, you are correct.

＞I believe we also agreed that there was no problem when I checked our sample API on my PC and asked Ninja Trader to check it.

I apologize for the error in my understanding.

I thought that IB Securities had asked me "which filtering is being applied" when I received the chat support in the past, so I asked NInja Trader about that and they responded.

I have attached Ninja Trader's response and the results of the most recent verification to this ticket, and will send them to you again.

We will also inform NInja Trader that there is no problem when IB Securities checks the sample API.

＞Please let us know what we need to check from the information you have provided.
Also, please could you use our sample API and compare it with NinjaTrader's tool to see if it is similar to TWS and different from TWS?

Yes, I would like to do that.
I will also try our sample API and will contact you again after confirming the results.


IB Securities

Thank you for the attached file.
We have received it from the previous correspondence and have confirmed it.
However, I am not sure I understand the intent of your question, so could you please clarify?
We are unable to investigate what is wrong on our side if TWS and Ninja Trader only have different descriptions.
Also, if Ninja Trader is correct, could you please explain to us what we need to confirm with proof of the correctness?


This is all I have to say at this point.

[ラリー]

We apologize.

The link from the stock search code does not seem to be working properly. We apologize for the inconvenience, but we would appreciate it if you could search for the stock code from the top page and check the chart.

Click the blue link under Stock Data Search>Local code>Name

https://www.jpx.co.jp/english/

[NinjaTrader_Jason]

As far as I know, the realtime data you receive in TWS comes from a different source than the realtime data through the IB API that NinjaTrader receives. I suggest to confirm this with Interactive Brokers. This likely causes the difference you report in regards to realtime data. The NinjaTrader platform receives realtime data as it is provided through the IB API. It only filters realtime data if 'Filter bad ticks' is enabled. It will then filter bad ticks as per post 7. It does not ensure that realtime data is the same as in you receive in TWS.

[ラリー]

Thank you for your reply.

As far as I know, the real-time data received by TWS comes from a different source than the real-time data NinjaTrader receives via the IB API. I suggest you check with Interactive Brokers to confirm this.

I will check on that point.


Filter settings, are my settings correct?

If they are correct, even after doing them, the Ninja Trader platform did not seem to match the TWS charts and prices in real time.




IB Securities has asked for clarification on the following points
We would appreciate your response. 1.

1. what is the problem with NinjaTrader and what is the problem with us?

2.Which points should we confirm with our developer?

3. what documentation (e.g. TWS and sample APIs, etc.) do you have on our side of the issue? (e.g., TWS and sample API have different price values, and the difference is displayed in Ninja Trader's tool as well).

The above.

We apologize for the inconvenience and thank you in advance for your patience.

[NinjaTrader_Jason]

Please note that there are no "correct" filter settings. It allows you to filter bad ticks on realtime data. It does not ensure that realtime data is the same between NinjaTrader and TWS.

You report a difference between realtime data received in NinjaTrader charts and TWS charts. Please check with Interactive Brokers the source of realtime data for TWS charts and if this is different from realtime data received through the IB API (which is what NinjaTrader receives). It seems point 3 confirms that TWS and IB API have different realtime data (different price values).

[ラリー]

Thanks for your reply.

You report a difference between realtime data received in NinjaTrader charts and TWS charts. Please check with Interactive Brokers the source of realtime data for TWS charts and if this is different from realtime data received through the IB API (which is what NinjaTrader receives). It seems point 3 confirms that TWS and IB API have different realtime data (different price values).

I will ask IB Securities about the above.

I will contact you again as soon as I receive a response.

[ラリー]

I have already tried various attempts at filtering but there seems to be no change and it was not resolved.

The API validation results (6627 instrument) shows a price match with TWS as attached.
However, the Ninja Trader platform does not seem to match the prices, graphs, etc.

Below is IB Securities' response in Ninja Trader post 13.

Thank you for your confirmation regarding the price comparison with TWS using the sample API.
My understanding is that the sample API provided by our company is positioned as the same tool as Ninja Trader's, and if TWS and the sample API match, then the sample API and NinjaTrader's tool should also match in real-time prices. I believe that NinjaTrader's tool should also be able to match the real-time price.
Why don't you give NinjaTrader the sample API and NinjaTrader's tool to NinjaTrader and ask them why the Realtime prices are different even though they are the same API?
Also, we would like to see evidence that the prices of TWS and the sample API were matched.

That's all for now.

The following is the content of our communication with IB Securities.
Some parts have been omitted due to the length of the document.
A translation has been used.
Thank you for your patience.


IB: In other words, which chart bars are different and which numbers do not match?
I: Frequently different charts or bars are one-minute or one-day chart bars with different opening prices for the day, and the high or low price for the day is often the highest or lowest price for the day.
IB: In such cases, the issue can be addressed and then troubleshooted, and I believe these can be resolved.
These are just downstream rendering issues.
Me: I see. How can I resolve this?
IB: If the problem is with the TWS source itself compared to the exchange source, then it is an upstream problem, not a downstream problem.
I would tell Ninja to adjust the logic.
Well, if the client has compared two different data types, i.e. querying bid_ask chart using Ninja but comparing with TWS trade chart, then it is not a problem but a way to use Ninja chart questions.
Me: Ok, I will contact Ninja Trader, but as a way to let them know, should I tell them the following?IB Securities has instructed me to adjust the logic of the Ninja Trader platform. I have also tried several times to initialize my computer and reinstall Ninja Trader to see if there is a problem with my environment, but this did not resolve the issue. Also, all historical chart graphs and prices in TWS and Ninja Tarder seem to match in real time as well when reloading on the Ninja Trader platform.
IB: No, I wouldn't tell them to forward the question, but when you use their API as a data source, it is their business to resolve discrepancy issues.
You are also their client, you are perfectly legitimate to tell your service provider to see if there is any problem.
If we are an upstream source and TWS itself has a problem compared to upstream, that is a problem to be solved, and in this case of course the downstream API software has the same wrong data. The question is how to identify the source point of the problem.
Me: Ok, I understand. I will ask NinjaTrader about the logic of the NinjaTrader platform and how to identify the source of the problem.

The content of the exchange is above.

Again, TWS and sample API prices were matched.

I will give you the sample API, my Ninja Trader platform backup file, TWS ID and password, and would you please verify this on NInja Trader's side as well?