NT8 1-Tick chart data errors just at just after the time of connection

[bboyle1234]

I've found that NT8's 1-Tick chart will load incorrect data at just after the time of connection.
Here's how I found it:
I created a CL 11-19 chart with 1-Tick bars, 5 days back, using the Default 24x7 template.
It had existing historical data and the chart loaded with that historical data.
Then, at about 08:57, I connected to the data source, CQG, in my case.

Figure1 (appears on the right in the thumbnails below) shows the chart moments the data connection has been established.
It shows that there is a tick at 08:57:01.664, which is missing from the recorded historical data.

Then, I disconnect and reconnect the data source. Figure2 (appears on the left in the thumbnails below) shows the extra tick no longer appears on the chart. After the disconnection and reconnection cycle, the chart now matches the recorded historical data. However, new data inconsistencies will now be appearing on the chart at just after the time of the most recent connection.

It appears that the chart almost always loads erroneous data at just after the moment of connection.

[bboyle1234]

I found this error by creating an indicator that waits five seconds after the chart has switched to "Realtime" and then makes a new BarsRequest and compares the data it received with the data on the chart. By introducing the five-second, delay, I eliminated the possibility that the issue is due to concurrent BarRequests as I had previously thought.

[NinjaTrader_Jim]

Hello bboyle1234,

Thanks for your post.

I have attempted to reproduce the behavior with CL 11-19 and opening a 1 tick chart, disconnecting, observing the last tick on the chart, and then reconnecting. After reconnecting, I'm not seeing discrepancies with data loaded. Looking for individual ticks is also a bit tedious, and I could possibly be overlooking something.

Could you share the script that you were using to test this and provide some steps I could take to run into the behavior on my end, preferably with a less volatile instrument? It will help me to look into this further on my end.

I look forward to your reply.

[bboyle1234]

Thank you Jim, I have attached code that can reproduce the test results.
Here is your procedure:
Clear all NT8 workspaces and open just one chart. I used CL 11-19.

On that chart, set the data series as follows:
-- 1-Tick (Last)
-- 5 days back
-- TradingHours = Default 24 x 7
-- Break at EOD = TRUE
Full data series settings are shown in an attached image

Right now, you are not connected to the data source.
Add the "DataChecker" indicator to the chart. It should run, and create output similar to the following, indicating that the chart bars match the BarsRequest bars.
2019-10-05 11:20:25.9373701 Both 995909

Next, connect to the data source (I used CQG) and wait for the chart bars to load and the indicator to output its results.
You might see output like this:

2019-10-05 11:20:25.9373701 Both 995755
2019-10-10 02:09:56.1660593 Reader1 1
-- 2019-10-10 02:09:56.1660593 52.51 1
2019-10-10 02:09:59.7460000 Both 5
2019-10-10 02:09:59.7460000 Reader2 1
-- 2019-10-10 02:09:59.7460000 52.51 2
2019-10-10 02:09:59.7500000 Reader1 1
-- 2019-10-10 02:09:59.7500000 52.51 2
2019-10-10 02:10:01.0850000 Both 8

In this output, there are header lines and indented lines.

The header lines relate to "sections" of tick data in which the compared bars are either equal or unequal.
The first field in the header line is the begining time of the section.
The last field is the number of ticks that are contained in the section.
The middle field can be either "Both", "Reader1", or "Reader2".
When it is both, it means that for this section of ticks, the chart bars are exactly the same as the BarsRequest bars.
When it is "Reader1", it means that the section contains ticks present on the chart but not present in the BarsRequest bars.
When it is "Reader2", it means that the section contains ticks present in the BarsRequest bars but not on the chart.

The indented lines give extra details of the first few ticks of the section. Fields are: Time, Price, Volume.

You will find that the data comparison shows that all data is equal except at the time immediately after connection.
I suspect that if you perform this test on a slow-moving instrument, you will not always get data differences therefore I don't recommend following your suggestion of choosing a less volatile instrument.

Thank you very much,
Ben

[bboyle1234]

I've updated the code slightly.
1. The BarsRequest no longer subscribes to realtime data (by not attaching the update event handler)
2. Miscellaneous code tidy and simplificiation

[bboyle1234]

Is anyone following this up at the moment?

[NinjaTrader_Jim]

Hello bboyle1234,

I am still following this ticket.

I am trying to move forward trying to reproduce the issue presented in your first post manually as there will be many cases where a bars request could not match the same bars request of the main chart due to the scripts lifecycle, and when each bars request is made.

I will keep you posted with my findings.

[NinjaTrader_Jim]

Hello bboyle1234,

I have attempted numerous times to catch this issue by connecting to a CQG based data feed, taking a screenshot of some ticks received with realtime data, disconnecting and reconnecting, and then comparing my screenshot to the what is available on the chart now. I was not able to find any visual discrepancies like you have demonstrated in your first post in any of my tests.

Getting back to the test script, I had mentioned that it will be very difficult/unlikely to get a script initiated BarsRequest to always match the chart's BarsRequest. This is because the requests are made at separate times considering the life cycle of the script and when the chart makes its BarsRequest. If the issue is that we are seeing an additional tick appear which is not recorded with historical data, we should be able to catch that case taking manual steps.

If you have any further insight for steps that I can take to hit this issue without the script, I will be happy to continue testing. We could also schedule a call if you would like to show me first hand. If you would like to do so, please email me at platformsupport [at] ninjatrader [dot] com with the text "Attn Jim 2302149" Please also include a link to this thread. Our regular support hours are 8:30am to 6pm US Eastern, Monday through Friday.

I look forward to doing anything else I can here to reproduce this issue.