Multiple crashes per day during development in VS

[misova]

Hi

NT8 crashes multiple time a day (generaly once per 1-2 hours), when Editor is open
and code is edited in Visual Studio.

Editor compiles files saved by the VS in the background
and VS reloads when it is changed by Ninja Editor.

This combination is quite unstable.
NT8 crashes the hard way = without any exception / popup / or log message.

It happens frequently - multiple times per day, and these crashes are are clearly related
to something in combination of Ninja Editor + Visual Studio + compilation in the background.

My experience is, that NT8 crashes, while I am writing in Visual Studio - that means, I do not interact with NT8 directly. But saving files in VS invokes compilation and maybe some call in NinjaEditor, so this can be source of the crashes... from my user point of view.

If the Ninja Editor is closed, the whole NT8 is much more stable and crashes do not happen.

This is how the crash looks like:



Edit after 2 days of random occasional crashes:
My intuition suggests, that the crashing is related to the process / interaction between NT8 and Visual Studio, where

• VS2015 saves the files -> which invokes NT8 compilation
• After compilation, NT8 changes the VS project files -> which invokes Reloading project in VS2015

In this context, most of the crashes happens.

Count of crashes per day:
19.11.2016 = 4 crashes
20.11.2016 = 7 crashes

[NinjaTrader_ChelseaB]

Hello misova,

To confirm, you are saving the script in Visual Studio (and not compiling), is this correct?

If you remove all custom 3rd party scripts, and ensure you are using the latest version (currently 8.0.1.0) are you able to reproduce the behavior when compiling a system indicator?

(I generally advise renaming the existing Documents\NinjaTrader 8\ folder and generating an new folder that is clean to test with)

When NT8 compiles, this does cause some code to be triggered such as from Addons.
I'd like to first ensure the crash is not being caused by a script.

[misova]

Hello misova,

To confirm, you are saving the script in Visual Studio (and not compiling), is this correct?

If you remove all custom 3rd party scripts, and ensure you are using the latest version (currently 8.0.1.0) are you able to reproduce the behavior when compiling a system indicator?

(I generally advise renaming the existing Documents\NinjaTrader 8\ folder and generating an new folder that is clean to test with)

When NT8 compiles, this does cause some code to be triggered such as from Addons.
I'd like to first ensure the crash is not being caused by a script.

Hi ChelsaB

Yes, I can confirm, that I am directly saving (Ctrl + S) files from VS. Compilation is done by NT8 automatically and in the background.

I will try to test with clean NT8 folder, without any custom scripts and write here back, if the same problem happens again.

[misova]

I have new finding - I clicked debug, after crash happened and this is exception message:

System.StackOverflowException was unhandled
Message: An unhandled exception of type 'System.StackOverflowException' occurred in System.dll

2 facts:

• Now, it crashed after 4+ hours of running NT8.
• In time when it crashed, I was writing something into VS2015 and not interacting with NT8 directly.

I expect, it does not relate to any recursive call in custom script. I believe this for more reasons:

• I do not use any external code / indicators / strategies addon-s. I use just my code and it is pretty simple = simple indicators, simple strategies, nothing else. These work completely stable for many years in NT7 and also operate in NT8 in a very stable way - never crashed, no problem at all, when using them. Any recursive call would be already detected and it would fail on each usage of such code. This does not happen at all.
• My deployed NT8 instance runs with all indicators and strategies deployed for a week without any crash, so it is quite stable. If there was problem with my simple custom scripts, it would crash reproducibly.
• This crashes happen during development only and almost everytime when writing something in VS2015.

Screenshot from TaskManager:
It does not show any excessive memory usage.




Screenshot from Resource Monitor:

[NinjaTrader_ChelseaB]

Hello misova,

(edit)
I'm wanting to determine if the crashes are occurring due to a script, or if this is due to an issue with NinjaTrader in general, or an issue with NinjaTrader within your particular environment.

To confirm, no custom scripts were in NinjaTrader 8 at the time of the crash, is this correct?

If there were, can you test using the steps from post #2 and let me know the results?

[misova]

Hello misova,

To confirm, you are saving the script in Visual Studio (and not compiling), is this correct?

If you remove all custom 3rd party scripts, and ensure you are using the latest version (currently 8.0.1.0) are you able to reproduce the behavior when compiling a system indicator?

(I generally advise renaming the existing Documents\NinjaTrader 8\ folder and generating an new folder that is clean to test with)

When NT8 compiles, this does cause some code to be triggered such as from Addons.
I'd like to first ensure the crash is not being caused by a script.

Hi ChelseaB,

• Yes, I am using the latest NT8 8.0.1.0 version.
• When I create new clean Documents\NinjaTrader 8\ folder I did not experience crashes. It works fine.
• But I have to tell, that with clean folder, there is not much to confirm about NT8 stability. It does not help to exploit existing bugs in handling of custom scripts by NT8 itself. It is quite realistic to expect, that this part of NT8 is probably not mature enough to handle various problems, when writing real-world custom scripts.
• NT8 needs to be stable and usable with real use-cases and real scripts. It is expected, it will work not only with with clean new folders but with custom scripts. That is the goal I think.

I just want to note, that crashes do not happen on some action (aplying indicator on chart) nor when something is running (like running backtest ). Crashes happen while typing code in VS2015, and NT8 in the background just crashes, without any interaction with NT8. That is good symptom, that something, that operates inside NT8 is not handled robustly enough.

I want to say, that I am not using any 3rd part code. Nothing form 3rd party.
I just have my own written simple scripts. Just clean NT8 + my own custom indicators / strategies.

I can make sure, that my scripts run fine. And they do. My strategies produce valid results in backtest analyzer. All indicators work fine and can be displayed on chart. No symptoms of any problem there. And the same strategies run fine on NT7 for several years.

But what can I do, when I am typing code in VS2015 and NT8 crashes in the background?

I expect, this is bug in NT8 or way, how it handles custom scripts. Or maybe it is something else... It is hard to know, what else can happen inside NT8, that can cause such occasional hard crash, that happens randomly.


I understand, how hard it can be to resolve these runtime crash problems. I don't expect any minute-fix. Maybe we should just watch, report problems, make notes about context and build a puzzle, until we have more clear idea, what or where to fix.

[NinjaTrader_ChelseaB]

Hello misova,

When using 3rd party custom scripts, it is possible for standard features fail to work as designed in NinjaTrader.

Below I am providing a link to the help guide on the impact of installing Addon-Ons.

http://ninjatrader.com/support/helpGuides/nt8/en-us/index.html?using_3rd_party_add-ons.htm#UnderstandingTheImpactOfInstallingAdd-Ons

I can say after the great many scripts I have written as examples (written through every state of the Beta and updated to the latest version), including indicators, strategies, and addons, I am not experiencing this crashing using the latest version of NinjaTrader 8 (8.0.1.0).

We are not able to prevent 3rd party code from causing NinjaTrader to misbehave if written in a way that it will do so.

Any scripts that you have written that are not official system scripts or reference samples written by NinjaTrader are 3rd party custom scripts.

The next step would be to try and narrow down which script is causing the behavior.
Generally, this is done by adding one script at a time and then testing for the behavior. Once the behavior returns, focus on testing the last added script by itself. If the behavior is still reproducible, comment out code in the script until the behavior stops, then slowly uncomment code until the behavior returns.

[misova]

Hi ChelseaB,

I found a problem and it explains, the "randomness" of crashes.
Crashes are not really random, but crash occurs after ~ 50 compilations of my project
and it takes only 7 minutes to simulate and reproduce.

Try to reproduce:

1. Open NT8 with Ninja Editor open (so it compiles automatically on source changes). Make sure, it is in Debug mode.
2. Open VS project and periodically edit (simply add or delete space) ---> then save by Ctrl + S ---> and wait until compilation in the background.
3. Check how memory increases by little step after each compilation.
4. After about 50 compilations, memory runs out of limits and NT8 crashes.

The randomness of crashes was not real randomness. It only took various time,
depending on how often I saved source files and when I reached this limit.
Sometimes it took 1 hour only, another time hours.

It is completely reproducible every time.

I recorded a video for you, so you can see, what happens:

1. how I start new fresh NT8 and how much memory it allocates.
2. then I do 50x compilation ---> while you can watch, how memory increases by a little bit after each compilation
3. until NT8 crashes on 47th compilation

Here is the video: https://youtu.be/LSNl-RxAysI (FullHD recorded)

To increase chances, of reproducing this, I have sent you my NT8 folder zipped, via email. (with subject: Attn: ChelseaB: NT8 crashes after 50 compilations).

I have bigger NT8 project with many new classes, so this problem can be more easily reproducible on real project, instead of fresh new NT8 project.

I tested and I can simulate the same behavior even on new fresh NT8, but with bigger project with many classes, the memory grows much faster = it is faster to reproduce.

[NinjaTrader_ChelseaB]

Hello misova,

I'm not quite clear at this point.

Are you able to reproduce the behavior without any custom scripts in NinjaTrader?

[misova]

Hi ChelseaB,
I tested the same behavior on clean NT8 without any custom scripts.

I compiled clean workspace about 100x, and see systematic increase in memory after each individual compilation.

After 60 manual compilations, memory allocation increased from 220 MB -> 320 MB = 100 MB after 90 manual compilations. I expect, somewhere near 850 MB, NT8 crashes. But it would require to sit for several hours to achieve crash on clean NT8. I hope you don't expect me to do this

I would recommend to simulate it with much bigger count of classes in project, to achieve the effect faster. I expect, that small count of classes fills up the memory much slower, compared to if you have hundreds of classes.

This behavior is reproducible, consistent and memory increase is completely linear - based on count of compilation started.

[NinjaTrader_ChelseaB]

Hello misova,

I am currently in the process of testing this behavior without any custom (self made or other wise) addons including bar types, indicators, or strategies within the Documents\NinjaTrader 8\ folder.

I'm not clear on part of the description of the steps to reproduce this situation. When you mention "I would recommend to simulate it with much bigger count of classes in project, to achieve the effect faster." can you clarify what this means?

[misova]

Hello misova,

I am currently in the process of testing this behavior without any custom (self made or other wise) addons including bar types, indicators, or strategies within the Documents\NinjaTrader 8\ folder.

I'm not clear on part of the description of the steps to reproduce this situation. When you mention "I would recommend to simulate it with much bigger count of classes in project, to achieve the effect faster." can you clarify what this means?

Hi ChelseaB,
I expect that problem can be in heap memory allocated for classes after each compilation. I am not sure, it is just my guestimate...

What I mean by "simulate with much bigger count of classes in project, to achieve the effect faster":
I expect, that the more classes the project has, the more memory it allocates after each compilation, what causes to reach limits for heap memory faster... This could explain, why the more classes project has, the faster the memory fills up.

So it is simple - if you have small count of classes, then memory fills up much slower, if you have many classes in project, then memory fills up much faster.

To have an example, NT8:

• in my project with all indicators and strategies (I have many), NT8 crashes after 50-60 compilations
• on clean NT8 without any custom scripts, it can take much more compilations - like several hundreds. But the trend of linear growth of memory is still clear in visible in both cases.

But to make it clear, I am not expect in .NET memory management. It is just my guestimate, what could fill up the memory after each compilation.

It is important to say, that memory fills up just by compilations. No indicator is applied or strategy backtested. Memory of NT8 increases just by compilations. If it reaches limit (something like 850 MB on my machine, then NT8 crashes.

[NinjaTrader_ChelseaB]

Hello misova,

I'm wanting to ensure that you do not have any custom scripts in the Documents\NinjaTrader 8\ folder. This would include indicators or strategies that are not applied to any charts or windows.

To add custom classes, you are referring to custom scripts that contain custom classes correct? Have you tested for this behavior without any custom scripts in the NT settings folder?

[misova]

Hi ChelseaB,

Yes, I want to assure you, that I deleted NinjaTrader 8 folder and ran Repair from installer to create new one fresh clean folder.

I can confirm, that permanent growth of memory happens on new clean NinjaTrader 8 folder too.
The memory grows by 100 MB for plus/minus each 80 compilations.

I just added spaces to existing source codes, and pressed Ctrl+S in VS2015. This causes saving file and NinjaEditor invokes compilation in the background. You can watch NT8 memory - it grows linearly as count of compilations increases.

By "add custom classes" I am referring to custom scripts that contain custom classes.
Yes, these can be Indicators, Strategies or any other utility classes.

[misova]

Hi ChelseaB,

after your own try and test of this issue, let me know if you can reproduce the issue please.

I am really interested, because if issue was identified, it makes me happy, that many hours
spent on this bug were worth it after all