Unable to Monitor 500 Instruments using IQFeed

[aksamyo]

I am having a problem monitoring 497 NASDAQ symbols through IQFeed. The problem is that I am getting 1 minute bars on only about 200 of the symbols (and not on all 497 symbols).

To prove my case, I wrote a small program that monitors the number of bars that each of the symbols receive. The program outputs the resulting statistics to the Output window every minute.

Loren, of DTN ([email protected]) spent a few hours helping me with my problem. I sent him my script and he ran it in his copy of NinjaTrader.
The problem he has is that NinjaTrader would hang after adding about 50 to 100 instruments. It might do another instrument after 4 to 8 minutes. All other trading software he uses on his machine works fine.

He is using NinjaTrader 6.5.1000.11 while I am using 6.5.1000.10.
We are both using the latest IQConnect. I have an 8MB pipe to the internet and so does he. We both have dual core and 2GB memory. Under Task Manager, NT uses 5 to 29% CPU and IQConnect uses 0 to 1% CPU. Nothing else running on the computer and the total memory usage is only 1GB.

Loren can not help me further because his NT hangs. Can you guys help?

Attached you should find the script (split up into 2 parts so that it is below 19KB) and an instrument list.

[NinjaTrader_Austin]

Hi aksamyo, I'll check into this and report back to you as soon as we figure out what is going on.

[NinjaTrader_Austin]

Hi aksamyo, I successfully downloaded and created a strategy using the scripts and instrument list you provided. I imported the list of symbols right into an instrument list.

First, I applied the strategy and sat and watched as it added each and every symbol. This took a while, as you've probably noticed.

Then I started a new Market Analyzer to make sure most of the instruments had data. After importing the instrument list into Market Analyzer, there were only 5 or so instruments with zero data. I also applied a SMA column to the MA, which worked fine.

As the strategy was running, it was gaining data for each instrument. After 50 minutes had passed (keep in mind I started the strategy after the market close), this is what your strategy reported to me:

15:25:09.3528321    InstrumentCount=    497    InstumentsThatHaveBars=    492    MaximumBars=    1    MininumBars=    349

My IQ Connection Manager is only reporting 493 for number of symbols, so with only one instrument missing, I'd say that it appears that everything is working fine for me. My system (including NinjaTrader) has been on and running since Monday morning and NT is currently using 489MB of memory. Also running throughout the test were Outlook, Firefox, Remote Desktop, iTunes, and Skype--a pretty busy computer. The system has 4GB of RAM and a Core 2 Duo T9300 at 2.5GHz.

The only differences between my setup and yours are NinjaTrader, RAM, and probably the CPU. Due to these differences, and the fact that the test went fine for me, I'd recommend updating to NinjaTrader 6.5.1000.11 and maybe consider using a system with more memory.

Attached, you'll find the output of the most recent print statement from your strategy. I've removed extra characters to save bandwidth. Also included is a screenshot of the Market Analyzer and my IQ Connection manager.

[aksamyo]

I tried it at 9 PM and got 180 instruments with bars. Sure, I can get 497 instruments with bars if I wait long enough. But shouldn't I be getting 497 instruments with bars after 1 minute? When I said earlier that I was getting less than 200 instruments with bars, I meant after waiting 10 minutes between 9:30AM and 10:30AM. I expected close to 497 instruments with bars after waiting 10 minutes.

I believe the problem is with DTN. I do not believe the problem is with my computer. As I said earlier, I had 1GB of memory to spare and the total CPU usage never exceeded 28%. You did better only because you waited 50 minutes and the DTN servers were not busy after hours.

Could you please try it between 9:30AM - 11:30AM for 10 mins so that we can compare apple to apple? I will do it again tomorrow.

By the way, where can I find the IQ Connection Manager?

I am thinking of moving to BarChart next month to see if I can get more instruments with bars. I am planning to subscribe to their $70 for 500 symbols. I wonder if you could try my script with BarChart and let me know.

[NinjaTrader_Austin]

Hi aksamyo, I'll test again sometime this morning.

If you're waiting just 10 minutes, are you sure every instrument has a trade within that timeframe? All the logic is contained within OnBarUpdate() in your script, so unless a trade occurs, there won't be an update for that instrument.

IQ Connection Manager is located next to the clock at the bottom of your task bar when you start the connection--as you select your IQfeed connection in NT, watch down near the clock and you'll see a new icon pop up. If you're unsure, you can just hover your mouse over the icons one by one until the tooltip says "IQ Connection Manager".

[NinjaTrader_Austin]

Hi aksamyo, I did a fresh install of NinjaTrader with all the default options, like load 15 days of minute data and such. A fresh install also means an empty set of historical data, so NinjaTrader had to pull all the data from the historical data servers.

It took roughly 14 minutes for the strategy (from the strategy tab, not from a chart) to add all the instruments.

Then I ran your test again and here are the results:

09:18:41.4814523    InstrumentCount=    497    InstumentsThatHaveBars=    488    MaximumBars=    1    MininumBars=    12

Attached is the output trimmed to the latest set of bar information.

I strongly recommend upgrading to the latest version of NinjaTrader 6.5.1000.11.

Also, could you check what version of IQ Connection Manager you're using? That information can be obtained by double clicking the IQ icon in the tray.

Everything seemed to be fine for me; I'm running these versions:

• NinjaTrader 6.5.1000.11
• IQFeed Version: 4.3.0.3

[NinjaTrader_Austin]

I've left the script running in the meantime and it is now up to 493 instrument. IQ connection manager is reporting 495 symbols.

Attached is the full log of your script zipped up.

[aksamyo]

Thanks for all your help. I will take your advice on the NT version upgrade.

This morning before I ran the strategy, I followed all the Performance Tips that was in the NT documentation. It definitely helped because I got 347 in 10 mins.

[aksamyo]

I thought this problem was solved but it really isn't. It has happened 3 Mondays in a row and it seems to only happen on Mondays. I know it sounds weird. I hope you can test it this morning because tomorrow it would be too late.

I send you my program and symbol list in the beginning of this thread. Please use it to test on your end. I was having this problem with IQFeed the last two weeks so I tried it this week with BarChart but it still does not work. The problem is that I am not getting even a single OnBarUpdate event on Mondays! I am monitoring 497 symbols but you can try it on 50 symbols (to make init faster) and it will still not get a OnBarUpdate event.

Desperately needing your help. Thanks.

I am using the lastest version of NT (.11).
I am using Type=Minute, 1 Exclude Weekend=True, Session begins 9:30AM Ends=4:00PM, Calculate on bar Close=true, Min.Bars required=400, Entries per direction=100, Entry handling=AllEntries, Exit on Close=false

[aksamyo]

I solved the problem myself. I just had to change the Minute Bars & Daily Bars to 5 instead of 2 (in the Options->Data->Chart default look back period).