7.0.1000.4 Access Violation Exception in db code

[nailz420]

Hi,

7.0.1000.4 NT is crashing with the following exceptions in the trace logs:

2011-03-19 20:36:52:546 (Market Replay Connection) Cbi.AccountUpdateEventArgs.Process: Account='Replay101' ItemType='InitialMargin' Currency='Unknown' Value=500
2011-03-19 20:36:52:776 in OnUnhandledApplicationException
2011-03-19 20:36:52:793 ********* exception trapped *********
2011-03-19 20:36:52:793 Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
2011-03-19 20:36:52:793 System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Data.SqlServerCe.NativeMethods.CoTaskMemFree(IntPtr ptr)
   at System.Data.SqlServerCe.Accessor.Dispose()
   at System.Data.SqlServerCe.Accessor.Finalize()

2011-03-20 02:41:05:522 (Market Replay Connection) Cbi.AccountUpdateEventArgs.Process: Account='Replay101' ItemType='RealizedProfitLoss' Currency='Unknown' Value=-5818.79999999987
2011-03-20 02:41:07:378 in OnUnhandledApplicationException
2011-03-20 02:41:07:422 ********* exception trapped *********
2011-03-20 02:41:07:422 Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
2011-03-20 02:41:07:422 System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Data.SqlServerCe.NativeMethods.CoTaskMemFree(IntPtr ptr)
   at System.Data.SqlServerCe.Accessor.Dispose()
   at System.Data.SqlServerCe.SqlCeCommand.CloseInternalCommand()
   at System.Data.SqlServerCe.SqlCeCommand.set_Transaction(SqlCeTransaction value)
   at System.Data.SqlServerCe.SqlCeConnection.ObjectLifeTimeTracker.Zombie(SqlCeTransaction tx)
   at System.Data.SqlServerCe.SqlCeConnection.Zombie(SqlCeTransaction tx)
   at System.Data.SqlServerCe.SqlCeTransaction.Commit(CommitMode mode)
   at System.Data.SqlServerCe.SqlCeTransaction.Commit()
   at Wilson.ORMapper.Transaction.Commit()
   at NinjaTrader.Db.Globals.DBThreadProc()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Most of the time it's the first kind of exception. I think you might have a race condition where one thread tries to close a db connection that's been already closed by another thread.

I was able to reproduce this with multiple instances (6 for example) of the attached strategy using Market Replay running at max speed on CL instrument, 10 Range bars. I also randomized the strategy parameters a bit using 10,20,30,40,etc for trailstopticks parameter.

Attached please find the strategy to reproduce (use multiple!) and a trace log. I removed a large portion of the entries from the middle of the log to reduce its size.

Regards,
Tim.

[NinjaTrader_RyanM1]

Hi Tim,

Thank you for the report and files. I did not run into similar issues using multiple instances of the strategy in market replay mode. First step I would take here is uninstall / reinstall your SQL installation. The following link can help with this procedure and ensure everything is up to date:

http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=e497988a-c93a-404c-b161-3a0b323dce24

The message in the trace file points to issues with memory, so if possible I would evaluate this same scenario using another computer.

Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

[nailz420]

I uninstalled ms SQL Server Compact, both 32 and 64 bit versions and deleted the directories where they were installed previously and then reinstalled from the link provided. So far I've had no crashes.

Thanks,Tim

[NinjaTrader_RyanM1]

Great to hear. Thanks for updating us, Tim.

[nailz420]

I was too quick to report this problem solved. After 1 successful test the problem came back. I was able to reproduce it on my quad core Win7 64bit machine and on my dual core Win7 64bit laptop. I even reproduced it after I completely uninstalled SQL server compact 3.5 sp2 from both machines. However, I was not able to reproduce it on a dual core WinXP sp3 32bit machine or using WinXP sp3 32bit running in a virtual machine (single core) on the quad core win7 64bit machine.

[NinjaTrader_RyanM1]

Thanks for the update. You should try next a repair .net install.

Please follow the instructions below to re-install the Microsoft .NET Framework on your PC.

* Terminate any applications running on your PC
* Go to the Windows Control Panel
* "Add/Remove programs" for XP users or "Uninstall a program" for Vista users
* Locate and uninstall the .NET Framework
* If your highest version is 4.0 or greater than do not continue with these instructions and let us know and we will follow up with further instructions
* If your highest version is 3.5 then proceed and uninstall this version and skip the next line
* If your highest version is 2.0 then proceed and uninstall this version
* Install version 3.5 of the .NET Framework which contains the 2.0 version which is required by NinjaTrader 6.5 (Download from here)
* After the install please check for updates to the .Net Framework with the Windows Update function

Note: Do not reboot your PC in between these steps as this can potentially create more issues that we are unfortunately not qualified to assist with.

[nailz420]

The latest .NET version on the Win7 machines is 4.0.30319.

[NinjaTrader_RyanM1]

Please follow the instructions below to repair Microsoft .NET Framework 4 on your PC.

Download and run the .Net 4 installer from below. http://www.microsoft.com/downloads/e...displaylang=en

Once ran you will receive two options if .Net 4 is installed.

Select "Repair .Net Framework 4 to its original state"

Click "Next" and follow other prompts

Once the setup application is finished it may ask you to restart your computer, even if it does not please do so.

Please retest your issue and see if it is resolved.

[nailz420]

"repaired" .net 4 installation per your instructions but I can still reproduce the crash

2011-03-29 17:45:30:894 in OnUnhandledApplicationException
2011-03-29 17:45:30:904 ********* exception trapped *********
2011-03-29 17:45:30:904 Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
2011-03-29 17:45:30:904 System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Data.SqlServerCe.NativeMethods.SafeRelease(IntPtr& ppUnknown)
   at System.Data.SqlServerCe.SqlCeCommand.ReleaseNativeInterfaces()
   at System.Data.SqlServerCe.SqlCeCommand.Dispose(Boolean disposing)
   at System.Data.SqlServerCe.SqlCeCommand.Finalize()

Tim.

[NinjaTrader_RyanM1]

Thanks for the continued updates, Tim. I'm checking with development on the next steps and will update this thread tomorrow.

[olingerc]

Just to let everyone know that the 64bit crashing is almost always due to a bad update/install of NT. Re downloading the setup and installing NT again solves the issue. That is if the crashing is caused by trying to use any of the advanced features in the drop down menus of NT.

I don't know why there is even a 64bit version. You cant even trade live with it. Charts don't amazingly load any faster, what tasks are actually being distributed among the cores? I have 6 cores active, the 32bit version seems to use all of them at some point or another. Same with 64bit. More charts open on the screen uses up more RAM than anything else.

[NinjaTrader_RyanM1]

Hi Tim,

Sorry, we are out of ideas here, so you may consider a system restore to a previous point. There is something unique to your machines or OS so this would be the last software change you would make before having to look at the hardware.

olingerc: Can you please create a new thread with any issues you're having with the 64 bit installation or send a note to support 'at' ninjatrader 'dot' com. If a new install works good but you start having issues later on, we need to investigate the changes that are happening between these two points.

[nicknamed]

any solution since?

Hi all,
we struggle with the same problem for about a year (we have first encountered it when we changed from beta 13 to beta 14), our script is way more complicated, testing it on like 5 different computers with different OSs. Sometimes 64bit, sometimes 32 bit. We think the error is linked with NT memory handling, so when we run market replay data (MRD) test, then on ZB it survives one year of MRD only when 30 or 60 minute timeframe, not smaller. on ES, it dies in few weeks to few months even on 60 Min. The period is flexible so not perfectly replicable. I do not wonder you was not able to replicate it on your machine, as you have to put a lot of stress testing assets settings to generate this error and then wait, sometimes it dies next month of MRD, sometimes in 8 months. It depends on number of charts opened with strategy running and timeframes in those charts - the smaller, the sooner the error emerges.
+on virtual machines NT behaves differently, it is slower and memory handling is also somewhat biased, we do not do test on virtual machines any more.
+our last log entry was :
3.4.2012 9:17:20|1|128|Exit on close handling for strategy 'EriaV04F08' started.
3.4.2012 9:20:20|1|128|Exit on close handling for strategy 'EriaV04F08' started.
3.4.2012 9:23:20|1|128|Exit on close handling for strategy 'EriaV04F08' started.
this time without any entry about accessviolation (yes, sometimes it is not written in trace and there is no record about crash)
as the error emerges always on/after exit on close/exit from position.
We had only one strategy with one contract running, so we wonder also, why this handling was executed three times.
We are preparing fake strategy that will generate this error and will send it in, but we are totally sure it is not influenced by SQL or .NET versions or installations, so if you think there is no other advice, tell us that and we will not waste time with fake strategy preparation.
Tim, have you find out why there is this error? Any solution? Please help.
N.

[NinjaTrader_RyanM1]

Hi N,
Please share a simplified script that shows the issue you're seeing, along with steps showing how to reproduce it. We're happy to give it a run here on our side.

[nailz420]

No. Still get crashes on running market replay and don't know why.