Chart stuck at "Loading Data..."

[KBJ]

Before I stopped & restarted NinjaTrader I decided to try doing a disconnect and reconnect of the datafeed (currently using CQG) and this eliminated the "Aborting" message dropped NinjaTrader's memory usage from 1.1GB to 832.7MB and caused the charts to repaint.

However, now I see two messages on the lower left side of the Control Center -- "Disconnected - CQG" in gray followed by a green "Connected - CQG". Strange.

And after about 10-minutes, more memory was given back (total use dropped to a more reasonable 262.4MB - I still have 10-charts loaded each with a number of indicators of varying efficiency.)

Doing the historical data downloads one-at-a-time (boring) seems to work OK - i.e., no additional hangs (and note, I did NOT restart NinjaTrader from the above posting, although I wouldn't want to do this with a system we were actually using to place live trades from.)

When I loaded the 6E 06-11, the memory use jumped up to 577.3MB but dropped back to 258.3MB when that download finished. Most of the other downloads didn't use more than 50MB or so. When I got to downloading the ES 06-11memory use went from 254.2MB up to 975.2MB at its peak. This is for downloading the prior 3-months of data. Apparently this download historical data is done all into a single buffer, and it doesn't get segmented at all, so it hangs when/if there isn't enough memory left to load packets that are coming in. And this part of NinjaTrader isn't smart enough to just abort the process at this point and give a reasonable error message.

There... I've debugged it for you (free service from me to you as usual - I used to get paid for this kind of stuff before I started "playing" with NinjaTrader - I hope you put it to good use and just fix it. You're welcome.) (And "no" I'm still not willing to move to Denver to work for $25/hour.)

[NinjaTrader_Brett]

Hello,

Thanks for that information and taking the time to post, I will forward it to development. As always if you run into a reproducible scenario please email into support intsead of this tread will allow us to hash down and go into specifics of your setup and diagnose from there as well if your willing to do so. I would love to get development a reproducible scenario for this so that we can continue to better the software.

Thanks.

[molecool]

Yeah, this is still broken. It's been several years now and NinjaTrader remains unable to release a stable version that loads your data without intermittent data corruptions or hang ups. Ridiculous...

[EddyEd]

Application is hanging by using tick Charts with simple stocks. It is very curious. If I choose for example a resolution of 200 Ticks, the application needs for reload 3-5 seconds. If I choose 100 Ticks, approx. 10 sec. If I select 10 Ticks, it needs roundabout 1-2 Min. If I choose 3 Ticks, 5Minitues and if I choose 1 Tick, at minimum 10 Minutes. During this time the NT is Hanging.
I use the Kineteck Datafeed.
This cannot be?
The same procedure with Multichart 7: 1 Tick = 3 Seconds!!!

Add: If I change an Indicator, for example the Moving Average, the aplication is also Hanging the whole time as I explained before

[NinjaTrader_Brett]

Hello,

The data is the same no matter what time frame chart you load for input. All tick charts use tick data.

Therefor it is not the data load that is causing problems most likely in this case.

What most likely is going on is that the lower granularity you have the more bars there are to load in the chart. When you load a chart indicators need to calculate on all the bars on the chart from the chart. Since there are more bars indicators take longer to calculate.

What indicators do you have applied to a chart?

If you open a new chart with no Indicators and switch charts do you see the same long load time on switching granularity?

-Brett

[molecool]

If you listened: Eddy already mentioned that he's using an MA on his chart. And quite frankly I have experienced the very same behavior with very simple setups. Fact is that MC kicks NT's ass when it comes to speed and reliability. Painting bars or indicators should be a matter of seconds - not minutes.

Quite obviously there is an underlying problem you guys have refused to acknowledge and/or fix for years now. Every time someone complains it gets ignored or rationalized in the very fashion you have done below. Having been a SWE myself for almost 20 years I advise you to start listening to your customers.

Look - trading is a constant arms race. And if anything but a simple MA or RSI indicator brings NT to its knees then you are going to lose everyone but the most simplistic retail traders. I have coded in NT now for several years and have learned to work around its limitations. The only reason I have stuck with it is because of me liking C# and that much of my stuff would have to be re-implemented. If there was a competitor on the horizon that offers C# integration and NT compatibility I would switch in a heartbeat due to constant data corruption and stability issues. I had hoped that NT7 would fix that but you guys failed at providing a more stable platform. Every other time I launch NT I have to go and clean the database (a scary feature in itself if you think about it). MC doesn't need that - it simply loads data and keeps it properly indexed.

My 2 cents - take it or ignore it.

Hello,

The data is the same no matter what time frame chart you load for input. All tick charts use tick data.

Therefor it is not the data load that is causing problems most likely in this case.

What most likely is going on is that the lower granularity you have the more bars there are to load in the chart. When you load a chart indicators need to calculate on all the bars on the chart from the chart. Since there are more bars indicators take longer to calculate.

What indicators do you have applied to a chart?

If you open a new chart with no Indicators and switch charts do you see the same long load time on switching granularity?

-Brett

[EddyEd]

You are right,
it seems that the problem are loaded indicators, even that they are simple sma or ema. If I load the chart without indicators it loads the data fast, but with a few indicators it goes into its knees.
Even in trading it cannot be that if you change the data interval of your chart, that the aplication hangs a long time. In this time you can loose a lot of money without it isn't the fault by yourself!!!
Thats critical

[NinjaTrader_Brett]

Hello,

I would need a reproducible scenario on my side with any default NT indicator causes a long load time.

The default indicators do not bring my NinjaTrader to a halt with the described scenario.

I suspect some custom third party indicator would cause issue here that is coded in a non CPU friendly way.

Finally, we already have this on our list molecool to increase performance. We are looking at great features to do this in our next major release of the software which unfortunately however is still sometime down the road.


-Brett

[molecool]

That is actually rather difficult to do. The 'worst' you can do is to load a bunch of instruments manually in your strategy/indicator and assign them to different time series. And even then NT should handle them well in memory - after all, MC does.

I don't think that anyone complaining about this uses anything crazy or performs complex calculations on each candle that would quite obviously bring NT to its knees. But given today's memory capacity and processing power available I need to maintain that NT is the slowest among all the systems I have tested. That and the constant data corruption issues are an ongoing battle.

Hello,

I suspect some custom third party indicator would cause issue here that is coded in a non CPU friendly way.

-Brett

[molecool]

This was promised for NT7 and you guys still don't get it. We don't want new features - at least most of us don't - what we need is a more stable platform. I would like to go for one month without having to either replace or reindex (i.e. fix) the database. How about putting that on top of your 'features' list? ;-)

Just saying...

Hello,
Finally, we already have this on our list molecool to increase performance. We are looking at great features to do this in our next major release of the software which unfortunately however is still sometime down the road.

-Brett

[gringa]

Work-around for hanging while loading data

Recently I ran into an issue with NinjaTrader hanging while trying to load data every time I launched the application. I tried deleting the cache, day, minute and tick database folders, but that did not solve the issue. I also tried resetting the instruments per the directions of tech support, but that had no effect either.

Fortunately, I was able to find a work around on my own, which leads me to believe that the .xml file for the workspace may be at fault. This makes sense, as xml files store data and formats, and are essentially databases.

Here's what I did: I moved the .xml file for the workspace that kept hanging (ncat VolBreakout.xml) from the 'workspaces' folder to another location. I then relaunched NinjaTrader. The Control Center showed the old workspace name, but did not hang because there was no data to load since the .xml file was no longer there. I was then able to open a different workspace, and it loaded the associated data without incident. I used NinjaTrader all day, then closed and reopened it; the latest workspace reloaded without any errors.

To further test my .xml theory, I closed Ninja, moved the original (problematic)workspace .xml file (ncat VolBreakout.xml) back to the workspace folder, relaunched Ninja, selected that workspace, and the program again hung while loading data. I left it open for several hours and it was still hung when I came back. I again closed Ninja, removed ncat VolBreakout.xml from the workspace folder, reopened Ninja, and was able to select a different workspace, which loaded without problems.

Because I have seen other posts question whether Market Analyzer could be a factor, I should mention that the problematic workspace did have a market analyzer window. I am running NinjaTrader 64-bit 7.0.1000.14.

Since the ncat VolBreakout.xml file won't load, I've recreated the workspace under a new name (without Markek Analyzer) and all is well.

[NinjaTrader_ChelseaB]

Hi gringa,

Thanks for your note.

What indicator columns do you have in the Market Analyzer in the ncat VolBreakout workspace?

Are you able to open the workspace if you are not connected to a data feed?

If you can open the workspace when disconnected, try removing the indicator columns. Then save the workspace as a copy (File -> Workspaces -> Save workspace as...). Then restart NinjaTrader and connect and load the workspace.

Does the workspace open without those indicator columns?

[gringa]

Hi Chelsea,

Thanks for the very quick reply and the great questions. I am using a proprietary indicator for columns for multiple timeframes in the market analyzer, along with some standard columns (bidprice, askprice, lastprice). I've attached a screen shot.

I disconnected and was able to load the workspace in question. One thing I did notice when researching your questions is that I had forgotten to enter the vendor's order number in the indicator setup parameters for the market analyzer, so I entered that for each related column. I then closed and reopened Ninja, and this time the workspace loaded just fine, so I wonder if my mistake of having forgotten that order number caused my data issues....

At any rate, since the workspace is now loading I'm unable to answer your remaining questions. Let me know if there is anything else you'd like to know, and thanks again.

UPDATE on May 8th 2014:

Again had data loading issue today when first launching NinjaTrader, so after a lot of messing around with the problematic workspace I was able to disconnect from my data feed, open the workspace and remove the indicator columns, and save the workspace as a new file. When I restarted NinjaTrader the workspace opened without the indicator columns in Market Analyzer.

[dline]

Hi Guys,
I too have the loading forever situation sometimes they do some times they don't ... they only load after disableing AVG.
May I have a step by step method of what to try to fix this
Thank you

[NinjaTrader_ChelseaB]

Hi dline,

If you are certain that the download and loading of charts is faster with AVG disabled, I recommend that you add the NinjaTrader folders in the following locations to the exclusion list for AVG.

C:\Program Files (x86)\NinjaTrader 7\
(My) Documents\NinjaTrader 7\