NT Crashes

[CriticalTrader]

I've been trying for several weeks to determine why and how NT crashes on certain restarts or when opening a second workspace; this has already been reported and logged as Ticket Number 1505351. I'm including it here on the forum in case others may have encountered this issue, have been trying to figure out why it happens, or have additional use cases under which it occurs.

Since I'm developing a rather feature-rich custom indicator, my suspicion has naturally been a resource (concurrency) conflict that I may not have been properly managing.

Using Visual Studio the error is always:
The program '.[..]NinjaTrader.exe' has exited with code -2147483645 (0x80000003).
There is no exception being raised or error logged to the NT trace or log files.

That said, I think I may have figured out the cause...
After removing the _Workspaces.xml file and all my workspaces, I started over and re-created 4 test workspaces, carefully building up the number of chart tabs and enabling various features of my indicator. As I followed this process, I would periodically close NT and restart it... all workspaces would open fine... no crashes....looked to be very stable.

While I'm normally in a process of writing and debugging code, this time I made sure I did not have my typical NinjaScript Editor open in any saved workspace. However, I did have a NinjaScript Output window open in one saved workspace... my indicator logs various trace information which is useful to watch print.

I then opened a NinjaScript Output window for each of my other workspaces, saved them, closed NT and restarted. NT crashed immediately when attempting to open the workspaces. It reliably crashed each time I attempted a restart.

I then removed the _Workspaces.xml file, restarted NT and (successfully) opened one workspace. When I attempted to open a second workspace, NT crashed. At this point, I again removed the _Workspaces.xml file, restarted NT and (successfully) re-opened the first workspace. I then removed its NinjaScript Output window, saved the workspace, then (successfully) opened a second workspace, removed its NinjaScript Output window, saved the workspace, and repeated this for all but 1 of the 4 workspaces. I closed NT, restarted NT, and it now reliably restarts each time (again). Only 1 workspaces has an open NinjaScript Output window.

In hindsight, when I first encountered this issue, I'm pretty certain not only did I have a NinjaScript Output window open in each workspace, but often had a NinjaScript Editor open as well. I have not yet conducted a similar test where instead of having a NinjaScript Output window open in each workspace, I have a NinjaScript Editor window open in each workspace... TBD.

[NinjaTrader_Jesse]

Hello,

Thank you for the post.

I wanted to check, does this only occur in workspaces with custom indicators loaded? Mainly I wanted to check if this also occurs in any situation where you are only using standard items in the workspace.

I have been unable to see this on my end using the steps I could pull from the prior email and this thread, but will continue testing. If you have a workspace that does not use custom items you can provide, that may assist in finding the problem. If custom items are in the workspace, they would not be loaded on my end and may skew the tests in this case.

I look forward to being of further assistance.

[CriticalTrader]

I have done no testing without my custom indicator attached to every chart tab in my tests.

If you are trying to reproduce this without my indicator, stress out your test setup with several chart tabs with a test indicator that verbosely prints to the output window....that may help.
I will also disable the output trace stream and see if that makes any difference.

When I get a chance, I will also try to record a video, if you think that might help.
I should be able to release a beta version of my indicator in a few days... currently running some additional stress tests.

[NinjaTrader_Jesse]

Hello,

Thank you for the additional details.

If you have not tested without custom items, I would suggest to only proceed after doing that.

We would want to confirm if this is custom code related or platform related as they are vastly different troubleshooting paths.

For a platform problem you can generally reproduce these type of problems without using custom code, only stock items are needed to create the problem situation and we could provide that information to development.

If this is related to custom code, you would first want to reduce the code used to the most minimal syntax needed to see the problem. Generally this includes that you debug the custom code to find the error point, and then create a new sample that demonstrates the error. We can use that sample to file a report with development to see if it would be expected or not based on the code used. Unfortunately any protected items are generally not useful for beta reports as we need to see the syntax causing the problem to be able to resolve it or know if it was expected that it causes a problem.

After determining if this is specifically related to the custom code used or specific to the platform, I would like to know the findings as that will assist in resolving the question.

I look forward to being of further assistance.

[CriticalTrader]

Jesse,

For a platform problem you can generally reproduce these type of problems without using custom code, only stock items are needed to create the problem situation and we could provide that information to development.

Your platform is very flexible and there are many use cases which can and should be created to test if the platform can handle stress; this is difficult if not impossible to do with "stock items". Comprehensive stress testing is not quite that simple (one of many reasons for an open Beta program) and is especially necessary to increase the probability of hitting potential resource conflicts in multi-threaded systems.

I understand the difficulty of you trying to reproduce the error and you are under no obligation to test the stress case I recommended in my previous post. If I have time, I will develop a transparent test indicator that spits out piles of Print statements concurrently and see what happens. It would sure make things easier if I could trap the event in Visual Studio....

Note, this is the code from my print trace utility (I recently added the lock, thinking it might be necessary to prevent a conflict... it hasn't helped):
private static object printTraceLock = new object();
private void PrintTrace(string message)
{
if (OutputTraceMessages)
lock (printTraceLock)
Print(TraceMessage(message));
}
private string TraceMessage (string message)
{
string expandedMessage = DateTime.Now.ToString();
expandedMessage += "\t\t"+this;
expandedMessage += "\t"+(sessionTemplate!=null?sessionTemplate:"" );
expandedMessage += "\t"+(Bars!=null&&Bars.IsTickReplay?"TickReplay":" ");
expandedMessage += "\t"+guid.ToString().Replace("-","");
expandedMessage += "\t\tState."+State;
expandedMessage += "\t\t"+message;
return (expandedMessage);
}

The fault may very well be a defect in my custom indicator, but it is sure does seem odd that there is a very definitive causal relationship with the indicator's use of NT Output windows....
I disabled all Print statements in each instance of the indicator (OutputTraceMessages=false) and sure enough it did not crash, even with an output window attached to each workspace. With the Print statements enabled in each instance of the indicator (OutputTraceMessages=true), and with only a single output window attached to one of the workspaces, it does not crash. Add another output window to any other workspace and it crashes.

If your development team has done all the stress testing it plans on doing, even perhaps after reading my post, then fine. I'm still quite impressed with the NinjaTrader 8 platform and have plenty of other things to keep me busy, not the least of which is more testing. Again, if I have time, I will develop an unprotected test indicator to see if I can reproduce it with something you can use and pass along to the development team.

[CriticalTrader]

That was easy.
Found some time ;-)

[NinjaTrader_Jesse]

Hello,

Thank you for providing the sample, this was necessary in seeing what was happening.

I do see that by using the lock specifically this toggles if the print works or not in my tests. I tried modifying the included sample to comment out the locks, the sample worked. I did also see the crash while using the lock.

Can you provide a description of why the lock was needed in this case? I can pass that along to development as well, for the time being I would not suggest using locks on the Print statements specifically as this seems to create a lockup situation.

I look forward to being of further assistance.

[CriticalTrader]

I only added the lock to see if that would be the solution which prevents the crash.
I originally assumed a lock should not be necessary and that potential resource conflicts would be handled directly by the platform. When I first encountered this problem in my custom (cdaVolumeProfile) indicator, I did not have the lock statement.

I commented out the lock statements in the PrintStressIndicator and am still getting it to crash.... sometimes have to wait a minute or so or do something....doesn't always occur automatically without the lock statement. Though, when it happens it seems to save it's faulty state and then happens on restarts or without clicking anywhere.
Is this perhaps related to the fact that on a crash /tmp/Startup.Profile doesn't get removed and should be?

I look forward to being of further assistance too ;-)