NT8 B11 - Application Crash on Market Analyzer Load

[sdonahue]

Hello,

In certain cases, NT8 B11 is causing a full crash of the application during the loading process of the Market analyzer with the following stack trace:

2016-06-10 13:04:56:774 *************** unhandled exception trapped ***************
2016-06-10 13:04:56:774 The process cannot access the file 'C:\Users\Shawn Donahue\Documents\NinjaTrader 8\db\minute\USDJPY\20160610.Last.ncd' because it is being used by another process.
2016-06-10 13:04:56:775 System.IO.IOException: The process cannot access the file 'C:\Users\Shawn Donahue\Documents\NinjaTrader 8\db\minute\USDJPY\20160610.Last.ncd' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalDelete(String path, Boolean checkHost)
at NinjaTrader.Data.BarsSeries.Save()
at NinjaTrader.Cbi.Instrument.<>c__DisplayClass197_0. <RequestBars>b__1(Object o)
at System.Threading.ExecutionContext.RunInternal(Exec utionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionCon text executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System. Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

The problem is clearly a threading access violation issue. However, there is nothing in the code for the custom MA columns that I can correlate with being responsible for this since I am obviously not doing anything to directly access the file in question.

The MA is a simple MA displaying the Forex pairs from the FOREX instrument list and 3 custom columns. Can you think of anything within an MA column which could potentially cause this to happen? Currently, I do some initialization (all on the same thread) within OnStateChange on State == Historical, and in so doing, I had to do a lock around the code block since it appears that the same background thread was handling more than one cell resulting in unpredictable state and behavior. Other than that, everything is pretty standard with no additional locking, no dispatcher invocation or additional threading.

Is there a different or more reliable state in which to perform this? I am suspecting that this may have something to do with it.

The main requirement for the initialization is that I need to have an initialized Bars object to pass to an internal data adapter for calculation and processing.

Thanks!

[NinjaTrader_ChelseaB]

Hi sdonahue,

The error indicates that NinjaTrader attempted to read a historical data file, however, another program is reading this file and has locked the file. As NinjaTrader was not able to open the file, this caused an exception and caused NinjaTrader to crash.

The most likely reason for this, is backup software (in particular we have had many traders have this behavior from Microsoft OneDrive) or antivirus software.

Do you have backup software or antivirus software installed on this computer?
Is OneDrive enabled on this computer?

[sdonahue]

Hi Chelsea,

This is on my development VM, and I have neither any backup software or antivirus installed.

Update: This in Win7, and OneDrive is not installed at all.

[NinjaTrader_ChelseaB]

Hello sdonahue,

Just to confirm, you unable to reproduce this behavior when using only Market Analyzer columns that are supplied by NinjaTrader, is this correct?

If this is the case, may I test your MA column script to see if the issue is with the script?

If this is not the case, please send an email to platformsupport [at] ninjatrader [dot] com, so that we may schedule a call with you to find the other program that is reading these files.

[sdonahue]

I should also add that the issue has happened no matter what state I use for the initialization, so Historical seems to be the best so far.

In my current environment, I doubt that another program is accessing the data file. Rather, it is most likely one thread is accessing it and locking it at the same time a different background thread attempts to access it, which is causing the exception. There is no other application on the PC that would be accessing it that I can think of, nor anything directly in my code that should be accessing or locking it.

[sdonahue]

I have not tried it with the built in columns yet. The issue is random and I just observed it for the first time last night when testing a fix for another issue and starting and restarting the program multiple times. Let me try it with built in columns and see.

[rmt81]

NT8B12 Crash on Enabling a Strategy

I have spent most of the day trying to isolate a problem that may be the same as reported by sdonahue.

The problem I encountered was that NinjaTrader crashed upon enabling a Strategy in the Strategies window. The error message in the trace and log files was that Last.ncd could not be opened because it was already opened by another program.

As sdonahue suggested, the other program that had locked the file was NinjaTrader itself. Although I had forgotten it long ago, it turns out that I had checked the option to "Record live data as historical". When I connected to IB as Market Data provider, it opened the data files for instruments in my Strategies window, even if not enabled. This can be readily seen by monitoring the presence and timestamps of the relevant files in Windows Explorer. Apparently, I typically enabled the strategies before NT opened the files to "Record live data as historical"; in that case, all was well. However, in some cases, Last.ncd was already opened before the strategy was enabled; that generated an unhandled System.IO exception that completely crashed NT.

The issue occurred in NT8B11 and B12 on Windows 10. It occurred with Interactive Brokers as the data provider, but not with Kinetick. It occurred only with "Record live data as historical" checked. I experienced the problem with the MASampleCrossover strategy running on the Sim101 account.

Thanks to sdonahue for suggesting that it was a different thread of NT that had locked the file.

Turning "Record live data as historical" off solved my immediate problem. However, this should be fixed as it will undoubtedly affect many other users, probably in a variety of seemingly unrelated ways.

[rmt81]

NT8B12 Crash on Enabling a Strategy

I meant to suggest that the easiest way to reproduce this is probably to check "Record live data as historical", add a strategy the to Strategies window, connect to IB, and then verify in Windows Explorer that Last.ncd is being updated and is locked (e.g. by attempting to rename it), before enabling the Strategy.

[NinjaTrader_ChelseaB]

Hello rmt81,

I've attempted to reproduce this behavior but so far I have been unable.

• 'Enable market recording for playback' is checked and enabled
• Connected to TWS 957
• Started SampleMACrossOver and allowed this to place trades to the Sim101 for a few hours now

Is there a particular amount of time that I should allow this to run?
Are there any steps I've missed?

[rmt81]

NT8B12 Crash on Enabling a Strategy

Chelsea:

On your screenshot, "Record live data as historical" is not checked. I would start with checking it.

I also would suggest that you verify that Last.ncd is locked before enabling the Strategy. That could be by trying to rename the file.

[NinjaTrader_ChelseaB]

Hello rmt81,

I am happy to retest with 'Record live data as historical' enabled, however, I am not certain what you are suggesting when you mention 'verify that Last.ncd is locked before enabling the Strategy. That could be by trying to rename the file'.
Can you further detail this action?
Are you manually modifying a historical data file while NinjaTrader is open?

[rmt81]

After a day two weeks ago of reproducing this behavior consistently, I have not been able to reproduce it since.

However, I did figure out the missing necessary condition for a file such as D:\Documents\NinjaTrader 8\db\minute\CSCO\20160817.Last.ncd to be locked before a strategy using that instrument is enabled.

The necessary condition is that there is an open market position in that instrument. When a data connection is opened, the appropriate Last.ncd file will be created (if it does not already exist) and locked. It seems that this may not occur until one or more ticks occur; e.g., during after hours trading, I observed the file not being locked until several minutes after the data connection opened.

I have not been able to reproduce the crash. I still suspect it is a matter of two NT threads not being aware of what the other is doing, as sdonahue suggested.

I think it is partly a matter of events on separate threads coinciding by chance. It seems that once the program crashed, it did it more consistently. This may be a consequence of breakpoints I had set; I don't remember what they were. I am slowly learning the nuances of debugging a multithreaded program the hard way.

Hopefully this information will be useful if others have related problems.

[rmt81]

ChelseaB:

Sorry, I meant to address your questions more directly in my last post.

When 'Record live data as historical' is checked and a market data connection is established, NT appears to subscribe to any positions for which open positions are detected. Then a file such as 'D:\Documents\NinjaTrader 8\db\minute\XOM\20160817.Last.ncd' is opened and locked. You can see the file created or its timestamp changed in Windows Explorer for the appropriate folder. You can test whether the file is locked or not by attempting to rename it in Windows Explorer; if the file is locked, Windows will tell you and not allow you to change its name. I believe this part is quite repeatable.

The problem I had two weeks ago was that when I tried to enable a strategy for XOM, the method that enables a strategy obviously tried to open the file that was already locked and that threw an unhandled IOException that crashed the program. My trace file clearly indicated that the file being locked was the cause of the crash.

Apparently, NT usually detects when it has an open connection to an .ncd file and uses that same connection. But, occasionally that fails and the failure throws an exception that crashes the program. And I do not yet know how to reliably reproduce that failure.

Hopefully, this addresses your question more fully.

[NinjaTrader_ChelseaB]

Hello rmt81,

1. Connect to TWS 957
2. Enable 'Record live data as historical' as well as 'Enable market recording for playback'
3. Manually enter a position on an instrument
4. Restart NinjaTrader and reconnect to TWS 957
5. Enable the SampleMACrossOver on this same instrument
   Allow to place trades for session

Are these steps correct?

If so, I'll give this a test today.

[rmt81]

ChelseaB:

The steps are correct, but before proceeding to 5 I would check that the appropriate Last.ncd file is locked; if not, there is no point going further.

The second half of 5 is unnecessary; if the strategy is enabled successfully, we have failed to replicate the problem.

The chance of replicating the problem is probably greater if you start with a list, e.g. Dow30, let several of them take positions before shutting down and then enable all of them at once. But yesterday, not even that threw the exception for me. I think some other condition is required, but I have not figured out what it is.

Thanks for looking into it.

On a different note, is there a Version 957 of TWS that works with NT? The last time I checked the web site, it still said 954.