Async Data Feed Problems

[GrumpyTrader]

I am using IQFeed with NT8 B5. Having run NT7 and NT8 parallel for a while now the main performance issue I have is how NT8 downloads data. I don't think the async download actually improves performance. I actually find it grinds the system to a halt most of the time or causes iqfeed to disconnect. It is particularly problematic when I am running a multi security backtest when my datafeed is connected. It will often hang in the middle (my guess is because a data issue).

[NinjaTrader_Matthew]

You could be right it's related to the new methods of requesting bars, but ultimately we'll need to be able to reproduce to look into further.

So I may run this scenario by development, can you please email us your log and trace files from a time period that the async download halted and or caused IQFeed to disconnected.

In your experience, does the frequency of the the issue increase when both NT7 and NT8 are running, at the same time? Or would that be hard to tell as you usually run both at the same time?

[GrumpyTrader]

Hi Matthew,
Having NT7 connected at the same time does increase the frequency of disconnects, but it still happens with just NT8. I've written similar code for a custom application I wrote and there are diminishing returns on the number of requests you make at a time. It is also related to the size of the requests (i.e. requesting 2 days of data instead of 180 days). It also seems to be related to request errors (no data for a particular security) where it throws a monkey wrench into the works. I'll send the log/trace files next time it happens.
Thanks

[NinjaTrader_Matthew]

Ok thanks - I'll do some testing on my end.

Do you notice any correlation with tick data, or do you see the problems when requesting minute data?

[NinjaTrader_Matthew]

Additionally, can you please monitor your memory usage as you make these larger requests? Are you finding a higher than normal amount of memory being used around the time you run into issues with it halting?

[GrumpyTrader]

Yes, I find it gets up to 5-6Gb of memory usage (I have 32Gb in machine). It gets incredibly slow and often I have to kill the process. For backtesting, I find I have to disconnect from the data feed to get it to complete.

[GrumpyTrader]

Tick data for sure. In fact, I've stopped using the download screen for downloading tick data. I just use a chart with the data source set to 180 days to get tick data and just switch symbols. The latest test was just using daily data on a large set of future contracts (about 40 covering all the major future markets).

[NinjaTrader_Matthew]

Hi GrumpyTrader,

Just checking in, have you seen instances of this behavior again? If so, do you have log and traces we can review?