Slowwwww data retrieval speed

[katiewc]

I have about 50 stocks on my market analyzer... the fields I have is 3 Technical indicators, net change, last price and day high and low. When I use yahoo and interactive brokers I find the data load time is so slow.. (and many times doesn't even load successfully!). Anyone else having a problem like mine?

[NinjaTrader_Ben]

Hello,

Each instrument that is on the Market Analyzer is like having a full chart up.

Please see if you have a "max ticker" error in your log tab. If you do:

This means you are pulling too much data from IB at one time. Please remove instruments from any Market Analyzers you are using, reduce the number charts and SuperDOMs you have running, and reduce the number of tickers you have up in TWS. Then shut down NinjaTrader and TWS, wait 10 minutes for your data status to change with IB, and then retest.

Also, try right clicking the Market Analyzer>Properties and reduce the "# of bars to look back" to decrease the load time.

You also may want to view this link:

http://www.ninjatrader-support.com/HelpGuideV6/PerformanceTips1.html

[billitin]

Tons of bugs

I'm using mere 3 securities in Market Analyzer (MA), have a hundred bars lookback and one RSI indicator over 14 bars. Here's what's buggy:

(i) If that'd be three charts instead, loading data from IB would take me much faster. Right now for just three securities it takes me over 80 seconds, compared to 10 seconds for graphs (!). I can't imagine what would happen if I'd put 50 there.

(ii) after loading data is complete, one of the rows is displaying RSI value of 138.58 . When restarted NT, the same value changed to 38.32, and one other value showed up blank, even after having updated all data.

(iii) After I got fed up with slow data update in MA for securities X and Y, I couldn't see charts for X and Y for any time period anymore, and needed to restart NT

(iv) When I had same security X both o a chart and on MA, and I closed the chart, MA gave an error and asked to shutdown NT. I restarted, now MA pops up right away with the same error message - shutdown NT. So now I'd have to reinstall NT...

The list could go on, it's just too long. I think in its present state MA simply doesn't work, and that's a pity because that'd be a very valuable tool. How long ago did you guys roll out this feature? Looks like it's still in beta-testing mode.

[NinjaTrader_Bertrand]

Hi billitin, thanks for the feedback - the Market Analyzer works well for many customers.

Dataloading speed is always individual and may vary from provider to provider.

It is also recommended to set the lookback period to the min. number you need to initialize your indicators properly, please see this Market Analyzer specific performance tips here - http://www.ninjatrader-support.com/H...manceTips.html

For the error message you report, please forward us your log and trace files referencing this thread - Thank you!

[billitin]

The slow data retrieval turned out to be caused by a connectivity glitch with IB.

There's another issue, with MA data caching. Let's say I have a chart of X on 5-minute bars over 1 day. I have it up to date, I close NT, reopen in 15 minutes, reconnect and the chart update would retrieve the missed 3 bars only. However, if I do same thing with Market Analyzer it retrieves data for all day long again. How do I make MA cache historical data, just like charts do?

[NinjaTrader_Bertrand]

Hi, please see this link - http://www.ninjatrader-support.com/H...ricalData.html

You chart will always be reloaded from your data provider, if your request contains the current day.

There is a difference between the local database and the cache NinjaTrader uses (managed by the .NET framework).

[billitin]

Waste of resources

Retrieving data for the same day is not too bad and I could live with it if it's inevitable. But one thing that I don't understand is how different timescales are optimized in MA. Let's say I make two Indicator columns in one MA with both indicators running on 20 bars each, but one uses 1-minute bar and the other one 1-hour bar; lookback period for MA is 30 bars. When I try to load the data for such configuration it gets quite tough because it seems like NT is trying to load 30x60=1,800 1-minute bars, instead of loading 30+30=90 bars (20 times less!).

This is such a glaring waste of resources that it's plain ugly. This way it just doesn't make sense to run indicators on two different time scales in the same MA, I'd rather have 2 different MAs to save my time and RAM (but then watching two MA lists instead of one is a pretty cumbersome task). I suggest that NT either removes per-indicator time scale selection and assign it to the entire MA, not to confuse users, or fix this optimization problem.

[NinjaTrader_Ben]

Hello,


Retrieving data for the same day is not too bad and I could live with it if it's inevitable. But one thing that I don't understand is how different timescales are optimized in MA. Let's say I make two Indicator columns in one MA with both indicators running on 20 bars each, but one uses 1-minute bar and the other one 1-hour bar; lookback period for MA is 30 bars. When I try to load the data for such configuration it gets quite tough because it seems like NT is trying to load 30x60=1,800 1-minute bars, instead of loading 30+30=90 bars (20 times less!).

>>NT builds the 60 min. bars from 1 min. bars. Thats just how it works. Be sure your lookback bars is larger than your indicator averaging period.

This is such a glaring waste of resources that it's plain ugly. This way it just doesn't make sense to run indicators on two different time scales in the same MA, I'd rather have 2 different MAs to save my time and RAM (but then watching two MA lists instead of one is a pretty cumbersome task). I suggest that NT either removes per-indicator time scale selection and assign it to the entire MA, not to confuse users, or fix this optimization problem.

>>Thank you for your suggestion. I will forward this to our development department for consideration. I am faily certain openning two different MA's will not help. Each 60 min. bar is built by 1 min. bars regardless.

[billitin]

Data caching problem

Yes indeed Day and Minute bars fall into two different categories, I overlooked that.

But consider the following scenario where caching data seems not to work:

(i) I connect to data provider, and load a list of instruments into MA. The MA template has option "exclude weekend" set to true. Now I've got the up-to-date market data because now it's Saturday afternoon at the stock exchange.

(ii) I restart NT, and load same instrument list on same MA template, without data feed connection.

(iii) I connect to same data feed now and... data start streaming, the entire process taking me a lot of extra time and RAM.

Clearly, there's no need to update data because last referenced bar is found only the previous day and I've got "store real-time bar data" ticked off. Why doesn't NT use data cache in this situation?

[zoltran]

We can only hope that the horrible way NT caches/reuses/re-requests data is fixed in NT 7.

I can't think of any other charting software that does it this way.

As you said, it's ugly, resource intensive and a real achilies heel with NT.