Does CQG have incorrect historical data? Who faced?

[EdLan]

The situation is as follows. NinjaTrader8, CQG connection.
On 12.21.2020, I download the maximum available history for contracts CL0121, CL0221 and save it in CSV format - 1 minute.

Yesterday, 12/25/2020, I completely update the history in NinjaTrader and save it again in CSV format - 1 minute. Then I compare the data with each other and find significant differences. For each contract, 15,000 - 19,000 record (row) mismatches only by OHLC! If you add the Volume field, there will be even more. Of course, no rolloving, and the date period is the same. It is noteworthy that there is an end date for discrepancies - 11/23/2020. All discrepancies before this date, after it - ok. What is this anyway?!
I know that data providers can change history, but I never knew it was that much! Differences in prices and volumes can be very significant. See screenshots.
What to do?
Change provider? Rithmic - isn't there? Or?

[NinjaTrader_Jason]

Hello EdLan,

Sorry I don't see how the 3 screenshots show the discrepancy. The 1st and 3rd screenshot show different expiry months (01-21 and 02-21) as well as different times (the date is the same). If you compare data from the same expiry month and the same date/time, do you see the same data?

[EdLan]

Hello EdLan,

Sorry I don't see how the 3 screenshots show the discrepancy. The 1st and 3rd screenshot show different expiry months (01-21 and 02-21) as well as different times (the date is the same). If you compare data from the same expiry month and the same date/time, do you see the same data?

Hello!

Please see line by line. Same contract, same date, same time, but different data OHLCV.
For example,
was:

CL0121

2020-10-27 04:13:00-05:00

39.02

39.07

38.95

39.03

175

became:

CL0121

2020-10-27 04:13:00-05:00

39.02

39.04

39.02

39.04

5

Perhaps it will be more convenient to look in the сsv files. Please see attachments. CL0121_combined_mismatches.zip CL0121_was.zip CL0121_became.zip

Thank you!

[NinjaTrader_Jason]

Can you please clarify how you obtain the historical data. Do you go to Control Center-->Tools-->Historical Data-->Load-tab-->Export or do you export it from the Edit-tab of the Historical Data window? When I export it from the Edit-tab, the Excel file uses a different format than your Excel files.

What time zone do you have selected in NinjaTrader at Control Center-->Tools-->Options-->General-->Time zone?

[EdLan]

Can you please clarify how you obtain the historical data. Do you go to Control Center-->Tools-->Historical Data-->Load-tab-->Export or do you export it from the Edit-tab of the Historical Data window? When I export it from the Edit-tab, the Excel file uses a different format than your Excel files.

What time zone do you have selected in NinjaTrader at Control Center-->Tools-->Options-->General-->Time zone?

I open the chart with the settings from the screenshot. Be sure to Reload All Historical Data from the drop-down menu on the chart.
Then I add a strategy (your strategy) taken from this forum that converts the data from the chart to csv format:
https://ninjatrader.com/support/forum/forum/ninjatrader-8/platform-technical-support-aa/1124281-export-to-excel?p=1124283#post1124283
https://ninjatraderecosystem.com/user-app-share-download/export-nt8/

Time Zone: Central time.

[NinjaTrader_Jason]

Can you please go to Control Center-->Tools-->Historical Data-->Edit-tab. Double click Historical-->CL 01-21-->Last-->Minute-->October 2020-->10/27/2020. Is the historical minute data you see here the same as your .csv file?

When I check the data on my end, it shows the same values for what you have for "became". I'm not sure why the strategy exported historical data differently before. If you export historical data at Control Center-->Tools-->Historical Data-->Load-tab-->Export on different days instead of using the strategy, do you notice differences?

[EdLan]

Can you please go to Control Center-->Tools-->Historical Data-->Edit-tab. Double click Historical-->CL 01-21-->Last-->Minute-->October 2020-->10/27/2020. Is the historical minute data you see here the same as your .csv file?

When I check the data on my end, it shows the same values for what you have for "became". I'm not sure why the strategy exported historical data differently before. If you export historical data at Control Center-->Tools-->Historical Data-->Load-tab-->Export on different days instead of using the strategy, do you notice differences?

I think it doesn't depend on the data conversion method. I found similar inconsistencies at Yahoo finance API. But there the difference was not so significant. By this I mean that the problem is most likely on the side of the data provider.

I am attaching the requested screenshot.
As far as I can see from the information in the screenshot, the data is the same as in my file "became".

Of course, the problem is that once we reload the data, there is no way we can restore the data that was before.
I only caught this error because I am converting and saving the data separately without overwriting it.

What also surprises me a lot is that the last date of the "corrupted" data is 11.23.2020. Although I first downloaded the data on 12.21.2020 and not before. Before December 21, I simply could not do it physically.

[NinjaTrader_Jason]

Can you please send me your log and trace files so I can further investigate

You can do this by going to the Control Center-> Help-> Email Support

Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

Please reference the following ticket number in the body of the email: Att Jason, 2894950

[EdLan]

Can you please send me your log and trace files so I can further investigate

You can do this by going to the Control Center-> Help-> Email Support

Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

Please reference the following ticket number in the body of the email: Att Jason, 2894950

Thank you, Jason!
Sent.