NT Reliability

[Harry]

I am now using NT for several months, it has wonderful features, and allows me to do all the programming and testing I like to do, but there is one major issue: Reliability!

(1) Sometimes after connecting, NT stops during the process of loading or saving backfilled data for an open chart. Aborting (after waiting and waiting) will sometimes result in a NT freeze while showing the popup menu "Aborting. Please wait". After closing NT via Windows Task Manager usually the data base is corrupted. Restarting NT will produce the Popup "Saving" again with no aborting possible as before. Only way to solve the problem is deleting and reinstalling NT and restore the data, data base repair will not work. (I am doing regular backups not to lose any work).

(2) When connected NT usually loads up all data as required. If I disconnect and restart NT later, often part of the data that was correctly displayed during the prior session is lost. There are arbitrary holes in the charts. These holes seem to appear for instruments with open charts only. This is not due to data supply via IB, NT simply loses its memory when closing down. (See chart attached, FDAX was correctly displayed an hour ago, now data between 9:00 and 11:14 has been lost). This is annoying during weekends, as charts cannot be updated. Holes will disappear again when connecting, and reappear in a different way, when starting NT again in a not connected mode.

(3) Regularly, when opening Currency or Currency Futures, NT will not display the lower panels of the chart. The only difference between currency futures and other futures are the four digits after the separating point..... Reloading the chart template solves the problem here, so it is definitely a minor problem.

(4) Remind daily data (supplied by IB) for IPE futures is delayed by one day, so also indicators using this daily data like the Pivots indicator in CalcFromDailyData mode would produce wrong Pivots.

I would really appreciate, if these bugs could be solved. No further complaints otherwise!

[NinjaTrader_Jessica]

1) This issue may be coming up again because you keep restoring the same faulty database each time.

You could do the restore and try a repair before anything else. If you still hit the same error please send me your trace and log files as well as your database to support AT ninjatrader DOT com attn Jessica.

2) This again points to the database as your data is stored in the database. Please see if the above course of action helps this as well.

3) You can send me your templates through Help--> Mail to support and I will be happy to test the template.

4) This you would need to speak with IB about as NinjaTrader does not dictate the data you receive.

I know there weren't alot of direct answers listed here but if you can follow the instructions and send me the requested items we can look at your issues more closely.

[Harry]

Hi Jessica,

Thanks for your answer. I have already discussed various of these problems, it is sort of summarizing my bad experiences with NT. I want to point out that otherwise I am happy and even impressed with NT, just these little problems that come again and again...

(1) When IB stops backfill (currently unavailable or multiple charts with different timeframes and lookback periods for the same instruments) NT sometimes freezes. Aborting is not possible then and NT needs to be closed via the Windows task manager. Data base is destroyed and cannot be repaired. This happens regularly, maybe twice per month. Annoying. What makes NT crash on backfill?

(2) Just checked FDAX again. Reloaded it. Disconnected NT. Checked all the charts. Everything was perfect. Saving and closing NT. Restarting NT: Now FDAX data between 8h00 and 11h14 today has disappeared altogether. Why? Repeated this several times, everytime data between 8h00 and 11h14 disappeared. Why does NT not save all the data properly? (When connecting the missing data will automatically refill, but during the week-end this is not possible...). Can you recreate this with data supply from IB?

(3) Lower chart panes disappear then and now for FOREX and currency futures, possibly because of the four digits after the point. Not linked to any type of template. Just happens. Not important.

(4) IPE futures data problem is due to incompatibility between NT and IB. IB's TWS displays charts correctly, so does AmiBroker. NT cannot display daily charts of IPE futures - one day missing, current day faulty. Old problem, just a reminder.... not really important, not using DailyBars for Pivots any more....

[NinjaTrader_Jessica]

1) You would need to send us a note with your files. I would not be able to answer this until viewing the file.

2) If you go to Tools--> Options--> Is store Real-Time Bar data checked or unchecked?

3) Any custom indicators?

4) I don't see any obvious reasons for this- Are there any errors under your log tab?

[Harry]

Different lookback periods causing the problem?

(1) I think I have a possible explanation for the freeze. This morning it happened again when loading backfill to my charts. I used a 2 min, 10 min and 60 min chart for the same instrument (FDAX). Freeze caused data base corruption. Had to restore data base from yesterday. Then I discovered that my charts had different lookback periods (I need at least 90 days on the 60 min chart for some indicators, but do not need as many days for 2 min charts, because it makes it slow, so I had a different lookback period on the 2 min chart). Now, if I set the lookback period for all charts to 90 days, the backfill loading works. I also had a similar problem recently when using the dailybars method for the modified Pivots indicator. This indicator would need similtaneous backfill for daily and intraday data, which also produced a freeze. So the freeze is possibly caused by two competing backfill processes for the same instrument somehwere between NT and IB:
- either backfill for two charts with the same instrument but different lookback periods
- either backfill for an intraday chart using an indicator that requires both intraday and daily data
(2) Real-Time bar data is checked
(3) Several custom indicators, all modified NT indicators.
(4) Old problem, look for previous threads with IPE. NT cannot use IB data correctly. Problem has been already confirmed.

[Harry]

Spending more time to repair than to use NT today

(1)I do not know what happens, NT crashed every time when loading backfill. Restore alone would not fix the problem, also had to remove and reinstall NT (latest version).

Also the behaviour of NT is mad: I have three charts connected to the DOM, all displaying the same instrument. If I change instruments, the popup Loading .. YM... will show up three or even four times, although all three charts use the same data (min, lookback period). And every popup makes me nervous, as I am afraid that it won't disappear while NT crashes.....

There is something in the way that NT handles the loading backfill process that is terribly wrong! Whenever I change instruments I am running the risk of a crash / complete reinstallation of NT.

[Harry]

Now crashing on daily data backfill

I am now watching NT try to save YM daily chart. Crash. "Aborting. Please wait..." is the last message I get again. This will be the 5th reinstallation of NT today. Always crashing on backfill. The worst is that the Aborting Option does not work either. Something terribly wrong with NT handling IB backfill.

[NinjaTrader_Eamon]

Hello Harry,

Thank you for your post.

I took a look at your trace and log files this morning. I couldn't find an instance of a crash within them. However, it did look like you were requesting too much historical data from IB.

The only errors found within the log files were:

"Max number of tickers reached" and "Maximum amount of historical data requested".

These are limitation of the IB data feed. The max number of tickers is due to too many instruments requesting data. This includes all instruments loaded into a Market Analyzer, Chart or SuperDOM / other order entry windows.

The Maximum amount of historical data is a limitation set by IB on requesting large amounts of historical data. IB limits your requests and will cause you to wait 5 minutes before you can request more historical data from their servers.

[Harry]

Major bug ?

Hello Eamon,

thanks for your answer. Indeed there is a problem between NT and IB. IB limitation is 100 tickers. I have 43 tickers running on TWS and my default list of NT(the only list I use) includes 43 tickers as well. This is together 86 tickers (IB counts both TWS and API tickers, even if same instruments) and therefore less than the 100 allowed. I only have one workspace, as multiple workspaces cause frequent crashes of NT because of backfill loading in the background that halts NT. The one workspace includes the control center, 1 DOM, 3 intraday charts, one daily chart and a market analyzer. The market analyzer includes 35 instruments of the default list. So I do not understand how I can, with this configuration exceed the alllowed 100 tickers.

I guess that NT fires multiple data requests into the API thus causing IB to cap its data supply. As NT shows the popup, it can be checked which instrument causes the problem. Today it was AUDUSD (market analyzer), YM (daily chart) and FDAX (intraday chart) for three different crashes I observed.

Besides NT being locked when the data supply for whatsoever reasons stops, it is by far the worst thing that

no aborting is possible! Trying to press the Abort Button on the popup crashes NT and corrupts the entire data base. This what needs to be changed.

If loading backfill stops and I can continue later on, who cares - but I hate deleting and reinstalling the complete Ninja Trader and restoring the last backup (sometimes manually copying my work into other directories before copying it back....)

Can you simulate this problem on one of your PCs? Does anybody on your side use IB data supply?

[NinjaTrader_Jessica]

Hello Harry,

The errors in your files are from IB. You data requests are simply to large for the feed you are using. There is nothing we could change in NinjaTrader that would have any effect.

Also the "crashes" you are referring to are not caused by NT. When NT crashes it writes what's called an exception to a file that we can review. It also give you a very specific error. This is not happening.

What is happening is your data requests is stallling TWS which in turn may be stalling your computer which may be stalling NT. I don't know what is freezing first but it all starts with the request to TWS for more data than you are allowed.

We have reviewed you logs a second time and still the only errors we can find are "Max ticker" and "Max historical data"

Also you do not need to reinstall everytime this happens. That is not needed, and if you are certain that the database is corrupt please send us a note and we can help you just drop the database.

Please let me know if I can be of further assistance and enjoy your weekend.

[Harry]

Sorry, the problem here is NT!

Hi Jessica,

first of all thanks for your answer. But I do not agree on your conclusion.

My data requests are not too large, because it normally works and I am well below the 100 tickers allowed by IB. I also saw the "maximum number of 101 tickers reached" message in the logfiles, but cannot explain it, as I did not change the number of instruments during the last weeks.

For the "crash" I should be more precise. First you will see the loading popup. If IB stops delivering data (sometimes backfill is not available for some instruments or it might be the 101 tickers problem) nothing happens. At this stage NT is still alive - you could check this under Windows Task Manager - but it can not be used! Now somehow you have to stop the backfill loading process, because otherwise you can not use NT!

The real "crash" occurs after pressing the "Abort" button. Now NT is dead, as can be seen in the Windows Task Manager, while TWS and all other programs are still alive and working. Now the only way to proceed at this stage is to remove the Control Center manually from the Windows Task Manager. After this removal the data base is corrupted and has to be restored. Sometimes also NinjaTrader needs to be deleted and reinstalled (repair would not work).

I am really sorry for bothering you, but first of all NT is a magnificient product, which is userfriendly and offers excellent execution and charting capabilites. For me it is near perfect, so I do not give up the issue! I just want NT to function properly. But something is definitely wrong somewhere in the engine. If NT does not get data, it simply stops and the"Abort" button then kills it. In case I use multiple workspaces the problem becomes worse (backfilling of other workspaces not being shown on the desktop), so I never do this.

I also used other charting programs (QuoteTracker and AmiBroker) with IB feed, and those do not get "locked", when the backfill stops. Please check again, I am pretty sure of what I am saying. So I would appreciate, if you really modified NT to avoid the "locking". I do not mind an incomplete backfill (can try again later), but I definitely do not enjoy the consequences of the "locking", which is data base corruption.

[NinjaTrader_VinceB]

Hi Harry,

Jessica is currently out of the office and I'll ask her to get back to you on Monday.

In the mean time have a great weekend.



Vince

[greentrader]

Hi Harry,
I looked into using IB for historical data and gave up on the idea after reading about all the problems people were having. IB severely limits (throttles) backfill requests to protect themselves.

NT Team,
I have a related question about performance... In my current setup, if I quit NT and restart it, it takes less than 30 seconds to reload all its windows and saved data. If I then open a data connection (outside regular trading hours), then NT appears to discard all the saved data and get new historical data (from IQFeed). This takes another 3 minutes, and serves no purpose since it's just reloading the same data. ("store real-time data" is unchecked) Why is NT wasting all this time reloading data it already has?

Also, does all this reloading (and not using "store real-time data") actually protect me from bugs like those that Harry is seeing? (i.e., by minimizing my reliance on NT's current database operations) I can live with this inefficiency (a premarket operation), especially if it's keeping me out of trouble, but am just curious as to what is going on...

(BTW, is there a faster way to "Save Workspace" than going through the hierarchical menu? Would also be nice if there was an option to auto-save the workspace periodically.)

[NinjaTrader_JoshP]

Please see this article: http://www.ninjatrader-support.com/H...HistoricalData

Thanks for the suggestion on Workspaces saving. We will put it on the list of future considerations.

[greentrader]

Please see this article: http://www.ninjatrader-support.com/H...HistoricalData

I had read that help topic, but I must not understand something, since I don't see how it applies to the case that I described: middle of the night on a Saturday, all US markets closed, NT restarted with all the data it needs cached, connect to IQFeed, and then it apparently throws out ALL its cached data and reloads it from IQFeed. What is the rule it's following that causes it to do that? It spends 3 minutes just getting back to the state it was in before connecting to IQFeed again.

Not a critical deal in this case, but I would just like to understand the logic being used so that I can minimize unnecessary historical reloads. Thanks.