Serious Issue with Unresponsiveness on version 8.0.11.1

[iantg]

On Friday while I was trading live I had quite an unpleasant experience. NinjaTrader 8 stopped working altogether. There was no live data being updated, my account position stopped updating, yet, some of the windows I could still move around in, so it wasn't a hard crash per se, I was able to close and disable my strategy and ultimately it ended up closing my position even though I couldn't see any updates. I had to use task manager to force close it because it was not responding to any attempt to close any of the windows. The error message was something to the effect of "Out of Quota"

When I tried to restart the program to ensure my position was closed and my strategy was turned off, I was not able to get it to start. I am enclosing the error message that was displayed. I emailed platform support and they observed that the program was unable to start, but they did not provide any insight into why it froze in the first place. For reference my ticket number is: NinjaTrader Crash Issue {1818119}

So I wasn't sure if this was an issue with the data feed, the server, ninja trader, or perhaps I was running it too hard (I was running an auto-strategy had taken around 14 trades in 20 minutes, and was printing output of every 75 tick bar update, as well as every row from the market data event handler).

So today, I started playing around in SIM / Market Replay, and I observed the same issue. I think after some testing I figured out the route cause of this issue and I would like to provide the following details, so that NinjaTrader is aware if they are not already.

I believe this issue is in refreshing the Trade Performance module while running beyond a certain threshold of resources on other tasks (Printing, trading, charting, etc.) With prior versions (< 8.0.11.1 ), I have been able to run at Max Speed on replay, send anything I want to the print window, and refresh anytime I would like. It would typically still run all the other processes but the Trade Performance module would take around 5-15 seconds, then it would update my statistics. In the newest version 8.0.11.1 if I am running it heavy (For example, printing output on 50 tick bar updates, trading a scalping strategy of 2 ticks vs 4 ticks, or 1 tick vs 3 ticks for example, and I refresh, the exact same thing occurs). There isn't a hard crash, but everything stops, the ticker stops running, no further trades are taken, and the the Trade Performance never updates.

When I was live, I was updating my Trade Performance module every 5 minutes or so, and this was what caused everything to stop running. The same behavior can be observed by doing the following:

In version: 8.0.11.1

1. Build a simple scalping strategy to ensure you get 100 + trades per day.
2. Run it on a fast time frame like 25-50 ticks
3. In the strategy have a print output running that sends something to the print window on every bar update, or like in my case on every change of market data.
4. Let it run for 10 minutes at max speed to get going using a certain amount of memory
5. Hit refresh on the Trade Performance while everything is running

You should observe the same behavior I did. I would consider this a very serious bug, because of what happened to me during live trading. I can imagine a few work arounds such as just not printing anything or printing less statistics, or running on a different time frame, minutes instead of ticks, etc. But this is bad overall and shouldn't require any sort of work around.

Let me know if you can recreate this, and if so, please let me know what you recommend? My guess is that users should just rollback their version to the previous stable version where this bug was not applicable. For example 8.0.6.1 does not have this issue.

Thanks

Ian

[pjsmith]

I think I might have just posted about a similar and perhaps connected issue, one thread up from you..... So many bugs.....

https://ninjatrader.com/support/forum/showthread.php?t=106563

[molecool]

Same here

Strangely enough I just came to the forum to report on the same thing. Is there a way to roll BACK to the previous version? It ran perfectly for me. I have a strategy with well over 8000 LOC with an embedded indicator that dons over 500 LOC. Both together pull 8 distinct symbols. I actually had it parse through a backtest spanning between 1/1/2014 until mid August of this year and it ran like a kitten on the previous version.

The current one is the pits, everything hangs, my CPU is running on 980% and the cooling fan on my MBP sound like a vacuum cleaner - apparently memory management seems to have gone to the dogs. Now we've been there many times over the years and with all due respect I don't want to spend hours submitting test sample code or whatever. Having to recode and relearn a ton of calls since NT7 was enough to bear and I just don't have the time.

Can I PLEASE just roll back to the previous version until you guys sort this out? Because that one was working perfectly and I want it back.

[iantg]

Good call on rolling back to a stable version...

I'm in the same boat with you guys... I think I am just going to uninstall 8.0.11.1 and reinstall one of my previous versions that worked fine. I am going to do this before I take anymore live trades on my server. I am sure NT support can eventually fix this, but I think NT should take 8.0.11.1 offline in the meantime and not allow any further exposure to this.

They should replace this version with the last stable version until they get this sorted out. I also advise that they communicate to their user base to anyone that has this version and notify them to replace it with a older stable version.

For anyone trading real money live, this is a disaster waiting to happen.

I am glad to help NT support get to the bottom of this, and hopefully get this resolved.

[iantg]

I have a couple different versions myself, but if you wanted to get 8.0.10 back (Assuming this was the version you had prior to 11 and it was working for you. Here are the steps:

1. Follow this link from the 11 release announcement: https://ninjatrader.com/support/foru...d.php?t=106344

2. Once you get to this link you enter your license key (SIM keys work fine) And it will take you to a download screen where it has the new version 11, but it also has the prior version 10.

3. From here just download 10.

4. Uninstall 11, but you can save your strategies, market replay data and anything else you have customized. Then just install 10, and you should be able to keep all your original strategies, market replay data, etc. If you need any help let me know, and I can walk you through it.

Ian




Can I PLEASE just roll back to the previous version until you guys sort this out? Because that one was working perfectly and I want it back.[/QUOTE]

[molecool]

Yes, I actually figured that out - more than happy to be able to roll back on this one. Next time I'll do what I usually have done with NinjaTrader updates: wait at least a month before I touch them.

[pjsmith]

Unfortunately, they seem to have updated the database on the last upgrade and it's not backwardly compatible, so I have to either remove/restore the db, or put latest unstable back in. The last truly stable version I had was 8.0.4.0 and I have support tickets for lockups on about every single version since then. 8.0.4.0 was solid and I ran in production for a long time with no issues. On later releases, if I went back to it, all was OK. Never versions have been unstable since in varying degrees with the latest version unusable for any real purpose.

Because of certain bug fixes that were required (data feeds etc), I can no longer run that version, but I would really strongly suggest NT dev team carefully review changes since then, because the current version is not fit for purpose. 80% of my dev time is spent dealing with system lockups and crashes. It's just not viable anymore.

[molecool]

Let's be honest NinjaTrader has been plagued by instability issues since 6.5. I was really hoping that the dreaded 'fix database' button would be obsolete in v.8 but obviously we are once again being lead down a long path of annoyances.

I literally talk to thousands of traders from all stripes on a weekly basis due to my trading blog which commands a wide audience from the humble retail rat to fund managers, and all the way to sophisticated/specialized quant traders. Very few are using NinjaTrader and I myself have stopped promoting the platform for one main reason: Anything beyond a simple MA cross over strategy is error prone and will repeatedly blow up on you. Which is the main reason why practically nobody in the professional trading community I know touches it with a ten foot pole. If you're a pro quant trader working for a firm and you have NT8 deployed please tell me how you do it because I cannot possibly imagine how you would justify that decision given the chronic problems NinjaTrader suffers from.

And let's not forget that a trading application should be developed like a NASA probe and not like an online game or texting app for the simple reason that it is mission critical. But we are repeatedly being forced to put up with DB corruption issues and every time we complain it's somehow our fault.

Instead of all those new/fancy features in NT8 all I wanted was a STABLE version of NinjaTrader. What we seem to being served again is not a stable rewrite of NT7 but a new error prone beast we need to tiptoe around and repeatedly repair in order to keep running.

And let me offer something else: I'm collaborating with a professional quant on various systems right now using quantopian and backtrader. Both run like a tank - rarely crash and there are no data corruptions. Both are pretty barebone but it's open source and if there's a problem you have a whole community working toward a stable fix. I believe that is the future and unless NinjaTrader gets its act together once and for all instead of focusing on being a data service, brokerage, and who knows what else, I will probably end up shifting my production systems over to those platforms.

Unfortunately, they seem to have updated the database on the last upgrade and it's not backwardly compatible, so I have to either remove/restore the db, or put latest unstable back in. The last truly stable version I had was 8.0.4.0 and I have support tickets for lockups on about every single version since then. 8.0.4.0 was solid and I ran in production for a long time with no issues. On later releases, if I went back to it, all was OK. Never versions have been unstable since in varying degrees with the latest version unusable for any real purpose.

Because of certain bug fixes that were required (data feeds etc), I can no longer run that version, but I would really strongly suggest NT dev team carefully review changes since then, because the current version is not fit for purpose. 80% of my dev time is spent dealing with system lockups and crashes. It's just not viable anymore.

[pjsmith]

I agree. I've started looking into similar platforms as well. I've a ton of time and money tied up in NT though. Thousands and thousands of hours of time and code.

NT just bailed on me again

2017-12-17 14:50:54:758|3|4|Strategy 'pjsDAXFT2': Error on calling 'OnStateChange' method: Object reference not set to an instance of an object.
2017-12-17 14:53:30:557|3|4|Strategy 'pjsDAXFT2': Error on calling 'OnStateChange' method: Object reference not set to an instance of an object.
2017-12-17 14:53:54:864|3|4|Strategy 'pjsDAXFT2': Error on calling 'OnStateChange' method: Object reference not set to an instance of an object.
2017-12-17 14:56:05:125|0|4|Unhandled exception: Object reference not set to an instance of an object.

Resulting in a 'debug' or 'close' dialog. Sigh.....

They'll say it's my code (and I think in this case it is as I'm coding away), but if you fail to validate any input and let your app crash as a result, I still think the app is at fault.

As it happens, I just did a repair database earlier this morning and things looked a little more sprightly for a while. Didn't last long.

Just to add - it was my code error. In the strategy, I tried to add an indicator to a chart with a null variable (I'd changed the variable name and not updated that line). But regardless, this sort of issue should not result in a hard crash where the entire platform fails. Of course the strategy should fail with the error noted, but 'debug' or 'close' is hardly graceful

[molecool]

I agree. I've started looking into similar platforms as well. I've a ton of time and money tied up in NT though. Thousands and thousands of hours of time and code.

Yeah, I hear you. Just one of my strategies has > 8000 lines of code. Migrating that from NT7 to NT8 was a massive undertaking, considering all the associated indicators and helper classes. To see NT8 turning as buggy (or worse) than NT7 really ticks me off.

NT just bailed on me again

2017-12-17 14:50:54:758|3|4|Strategy 'pjsDAXFT2': Error on calling 'OnStateChange' method: Object reference not set to an instance of an object.
2017-12-17 14:53:30:557|3|4|Strategy 'pjsDAXFT2': Error on calling 'OnStateChange' method: Object reference not set to an instance of an object.
2017-12-17 14:53:54:864|3|4|Strategy 'pjsDAXFT2': Error on calling 'OnStateChange' method: Object reference not set to an instance of an object.
2017-12-17 14:56:05:125|0|4|Unhandled exception: Object reference not set to an instance of an object.

I think I can help you find the LOC. First up put your NT editor into debug mode (right click).

Then wrap the code inside your OnBarUpdate method into this try/catch statement:

try {

.... your code

}  catch (Exception e) {
	Print("\nOnBarUpdate ERROR:");
	String error = Time[0] + ": XXX ERROR during OnBarUpdate():\n " + FlattenException(e);
	Print(error);
	Log(error, LogLevel.Error);
}

Finally add this method to your strategy:

/// <summary>
/// Prints the entire stack trace plus the exception message.
/// </summary>
/// <param name="exception">The thrown exeption to be printed.</param>
/// <returns>A string containing the exception message plus the stack trace.</returns>
public static string FlattenException(Exception exception)
{
	var stringBuilder = new StringBuilder();

	while (exception != null)
	{
		stringBuilder.AppendLine(exception.Message);
		stringBuilder.AppendLine(exception.StackTrace);
		exception = exception.InnerException;
	}

	return stringBuilder.ToString();
}

This will give you the line of code that's causing the issue. Over the past year or so this function has probably saved me weeks of hassle and wasted time.

Good luck!

[NinjaTrader_PatrickH]

Hello iantg,

Thank you for your post.

We will investigate your report based on the detailed steps you have provided. I will also follow up with you on your ticket so we may keep both this thread and your ticket updated.

I look forward to assisting you further.

[molecool]

Fortunately I still had the older version of NT8 running on one of my VPS' which allowed me to copy the old database over to my other system. Lesson learned: Always make a full copy of NT before upgrading!

Unfortunately, they seem to have updated the database on the last upgrade and it's not backwardly compatible, so I have to either remove/restore the db, or put latest unstable back in. The last truly stable version I had was 8.0.4.0 and I have support tickets for lockups on about every single version since then. 8.0.4.0 was solid and I ran in production for a long time with no issues. On later releases, if I went back to it, all was OK. Never versions have been unstable since in varying degrees with the latest version unusable for any real purpose.

Because of certain bug fixes that were required (data feeds etc), I can no longer run that version, but I would really strongly suggest NT dev team carefully review changes since then, because the current version is not fit for purpose. 80% of my dev time is spent dealing with system lockups and crashes. It's just not viable anymore.

[molecool]

Narrowed down performance culprit by accident.

I figured something out by accident that may help NT support to fix the current performance issues we have been *****ing about here:

I am using git to push any changes from my dev system to my production system. When doing an update the default indicators and strategies were pulled as well, which were incompatible with the older version. I still had a vanilla copy and thus replaced the new default indicators (causing the issue) with the older ones, so that I could compile. I didn't bother moving the @Strategy.cs class however (as I'm lazy).

Well guess what - after relaunch I realized similar performance issues on my production installation - and again i was running 8.0.10.0 there. On a hunch I shut it down and then replaced the new (8.0.11.0) @Strategy.cs file with the older one (8.0.10.0) in my Strategies folder.

Relaunched Ninja and voila - it's humming again! So, whatever is causing at least my performance issues is related to the new Strategy class. I suggest you focus your attention there.

[lmatiukas]

picky backing on memory consumption and code

1. It does seem like the code has to be perfect or else it crashed, so I go around that issue by not making any mistakes, and I can deal with it. Debugging, of course, is impossible by using the "debug" option provided when it does crash.

2. The memory consumption of last update 8.0.11.1 has been insane. I have 32GB ram, and just having charting open will go up to ~14GB, and so I cannot have anything else open really =/. Time for an upgrade I guess.

3. NT7 seems to always be stable, but all my code is in NT8, including some custom indicators. If you google for help converting it's all about how to go to NT8 FROM NT7; so if I'm stuck on a conversion, there aren't any resources to find how to downgrade =(. And NT7 doesn't use a real programming language, so what would make sense to me, somehow will give me an error, and then refer to point 1) of the program just crashing.

[molecool]

1. It does seem like the code has to be perfect or else it crashed, so I go around that issue by not making any mistakes, and I can deal with it. Debugging, of course, is impossible by using the "debug" option provided when it does crash.

2. The memory consumption of last update 8.0.11.1 has been insane. I have 32GB ram, and just having charting open will go up to ~14GB, and so I cannot have anything else open really =/. Time for an upgrade I guess.

3. NT7 seems to always be stable, but all my code is in NT8, including some custom indicators. If you google for help converting it's all about how to go to NT8 FROM NT7; so if I'm stuck on a conversion, there aren't any resources to find how to downgrade =(. And NT7 doesn't use a real programming language, so what would make sense to me, somehow will give me an error, and then refer to point 1) of the program just crashing.

Please follow my instructions in comment #10 and you'll save yourself a ton of time and frustration. If you have questions just ask.