Corrupted Tick Data IQFeed Downloads

[ChainsawDR]

Hi,

I've posted this issue on the iqfeed forum (http://forums.dtn.com/index.cfm?page=topic&topicID=5905) as I'm not yet sure if the issue is with NT8 or IQFeed. When downloading historical data from iqfeed using NinjaTrader 8, I'm frequently seeing issues with the tick data (unsure if its a problem with iqfeed, NT, or user error). Yesterday I set off a download of ~6 months of tick data for ~500 stock symbols, and today when testing using that data there were tick data issues with ~50 of them (obvious issues such as having 1MM trade volume for the same microsecond time stamp). What I have been doing as a workaround is to delete the data for the affected symbols and redownloading them. However when I just did this a moment ago, I'm still seeing the same issue for some of the symbols. A good example is the stock GPS...

I setoff a download for GPS Tick, Minute & Day data for Bid/Ask/Last from 4/1/21 to 9/30/21 (in NinjaTrader>Import Historical Data>Download).
The tick data for 9/13/2021 looks corrupted (minute data looks fine).
The screenshot attached shows tick data and 1min data side by side for GPS. In the tick data there are ~2MM+ trades in GPS with the timestamp 3:57:00:438, which 1min data for 3:57 has ~88k volume.

As deleting and redownloading has mostly worked in the past, my best guess is that it won't be a problem with iqfeeds data, but some other issue. Do you have any ideas what could be causing this please? (it's causing a lot of delays to identify, purge and redownload historical data). I'm wondering if there are alternative methods I could use for downloading and importing the data which could either bypass the issue or help debug it? FYI, I am backtesting strategies against a basket of ~500 stocks for ~6 months, so I download the historical data in advance so the tests run faster.

Many thanks

ChainsawDR

[ChainsawDR]

In case it helps, these are the stock symbols (all nasdaq or nyse) that were corrupted during the overnight download:
IOVA
GPS
SAVE
EQT
DISCK
KEY
SM
NWSA
EQH
IVZ
HUN
DVN
SPWR
AES
NI
NVTA
VRT


I then deleted all historical data and redownloaded for these symbols, which fixed the issue for all except the following 5:
GPS
DISCK
KEY
IVZ
OXY

I've just noticed that for GPS, the corrupted data on the original download was for different days (7/30 & 9/23 had corruption, not the 9/13 corruption from the second download) so I do think it'll likely be a NT8 issue.

Many thanks

ChainsawDR

[NinjaTrader_ShawnB]

Hello ChainsawDR,

Thank you for your post.

Could you please send the log and trace from your platform to look further into this inquiry?

To send me your log and trace files from within NinjaTrader. Go to Control Center-->Help-->Email Support in NinjaTrader. Put '3352061 - Att Shawn', in the subject field and enable 'Log and trace files'. Also, give a reference of this thread in the body of the email.

We look forward to assisting you further.

[ChainsawDR]

Thanks Shawn - just sent now

[ChainsawDR]

FYI, IQFeed looked into GPS and find no issue with the data from their side: "Checking the IQfeed data for GPS our tick data matches what you have for your minute data. Random check of some of the other symbols you listed does not show any issues. There have not been any reported issues with the listed symbols. You can check the data I referring to before it is transferred to NinjaTrader On your desktop select the icon IQfeed applications then select time and sales. In request details select tick data in settings you can select date specific data. NinjaTrader support should be able to assist with how their chart is building the data."

[ChainsawDR]

An additional question please: when the corruption of the tick data happens, the test run very very slowly (as can now be millions of ticks vs tens of thousands). I'm normally able to spot the issue because my backtests throw up an inordinate amount of trades in certain stocks, however in many cases no trades are taken so the data issue isn't flagged and I can't spot them. My backtest for May 2021 is running very very slowly so it feels like one of the instruments has a data issue that is slowing everything down. Do you have any tips for debugging? I was thinking of writing a tick count and writing it to streamwriter output, but not sure if this is the best method for identifying the issue.

Thanks again

ChainsawDR

[NinjaTrader_Heath]

Hello Chainsaw DR,

Thank you for your updates.

I've spoken with our team and can confirm that the data your receiving from IQ Feed isn't manipulated on our end at all, The data they provide is the data that is used.

I speculate it's more likely that the issues you're seeing with corruption are due to the amount of data you're requesting and the relatively small size of these data packets. Some packet loss is generally expected when transferring large amounts of small instances of data. And the variation in which data you're seeing as corrupted would reinforce that this is due to packet loss.

However I'll be testing this after hours as well to see if i can reproduce. Can you provide more information on the strategy that you're using and the settings you're using for your backtest so I can accurately reproduce this test?

Is the strategy itself purchased or publicly available? Is it one you have coded yourself?

Thank you I'll await your reply.

[ChainsawDR]

Thanks for looking into this Heath. The strategy I'm using is one that I coded myself, however I'm downloading the data via Import>Historical Data in one step, then running the strategy in another step. As it's unlikely that the strategy is causing corruptions of the already downloaded tick data, I suspect the issue is packet loss as you've described. If it's inevitable that this happens when downloading large volumes of data, is there any method that I can use to diagnose which instruments are affected? (I can't click into every instrument, for every day and every hour to see the tick count). For example, in an ideal world I'd run a SQL Query against the Db of data to sum the number of ticks per instrument per timestamp, then I'd sort high to low and review to find the obvious cases where there are millions of ticks for the same period, delete them and redownload. Any ideas like this?

Otherwise, I'm thinking my best option may be to download the data into CSV files from IQFeed directly, then import them into NT8. This would then confirm the packetloss scenario (as the csv's would be a distinct record of the data). I'm hoping to avoid this approach as it involves re-downloading all of the data over several days.

Thanks again

ChainsawDR

[ChainsawDR]

An update from my side: found somewhat of a workaround... if I navigate using File Explorer to /NinjaTrader 8/db/tick/ and the use the search feature for: "size: > 5 MB" this will bring up the prime suspects of instruments with corruption issues. I can then manually delete the affected files one by one and re-download them. For example...

I did this and found that for CCL, the Tick file for 2021/10/27 was 30 MB in size.
I then explored the Historical Data for that day and found that 9am was showing 10,133,713 records.
I deleted CCL, redownloaded, then the same day is now showing 9am with 11,735 records. (screenshot attached)

It's going to be a royal pain as file explorer is showing that there are over 50 files with suspected corruption but hopefully it'll only be an exercise needed whenever there's these large scale downloads (and backtest will now run much faster once purged). It'd be ideal though to have an inbuilt corruption mechanism within the NT8 software though, e.g. if you find there are 1,000 tick records with exactly the same millisecond timestamp (aka outside of the realms of possibility), to flag them as corrupt. Is it possible to log this as a feature request please?

[NinjaTrader_ShawnB]

Hello ChainsawDR,

Thank you for the suggestion.

I have submitted this as a feature request to the Development Team. I will follow up with an internal tracking number for your reference as soon as it is created.

Thanks in advance for your patience.