Exceptions & memory issues

[Pete S]

I've been running some heavy duty strategy optimizations over the past few days (i.e. overnight runs that last several hours) and have noticed a few things.

1. Memory usage gets quite high and does not seem to be freed up properly. I see OutofMemory exceptions in the log. I am going to upgrade to 4GB this weekend -- I suspect this is a .NET issue and there's not too much you can do about it, but are there any tricks people use for memory management, any hidden command line parameters, or such things?

2. Several features have been flaky. Saving optimization results has crashed about 25% of the time, and when I go to run a strategy afterwards I see exceptions about accessing Disposed objects. These things do not only happen after receiving OutOfMemory exceptions.

I'll provide more detailed information on these if they are not known issues after I add memory to my workstation.

[NinjaTrader_Ray]

Hi Pete,

1. NinjaTrader for sure relies on .NET garbage collection to clear out unused objects that eat up memory and the .NET garbage collector has a mind of its own on when and how it does this. Since you can virtually code anything you want within our strategies, please double check there that your code is not doing anything that could consume a lot of memory. For sure addtional RAM can help.

2. Once you have run into a memory exception, any unexpected behaviour after that is very possible. Does the exception on saving optimization results happen prior to a memory exception? If yes, any idea on how to reproduce?

[KBJ]

I found that if you use Print statements in your strategy, that will cause memory to be used up faster.

For some of my strategies where I was trying to make sure I wouldn't run out of memory during optimization, I changed the code to write the data to an output file instead.

For instance,

using System.IO;
        

static private StreamWriter IO_Pointer = null;  // Used to write the log file.


//--------------------------------------------------------------------------------------------------
private StreamWriter Create_File( string fileName )
{
    FileInfo f_info = new FileInfo( fileName );
    
    if (!f_info.Exists)                            // Does the file exist?
    {                                              // No.
        StreamWriter fptr1 = f_info.CreateText();  // Create a new file.
        return( fptr1 );                           // Return file description.
    }
    StreamWriter fptr2 = f_info.AppendText();      // Open existing file.            
    return( fptr2 );
}

        
private void Write_File( StreamWriter filePointer, string line )
{
    if (filePointer != null)                       // Was an open done yet?
    {                                              // Yes.
        filePointer.WriteLine( line );             // Writes out a line to the specified file.    
        filePointer.Flush();                       // Empty the buffers by writing them to disk.
    }
}

//=====================================================================================================================
//  Dispose -- Do cleanup (opposite of Initialize method)
//
public override void Dispose()
{
    // The following cleanup is done at the end of each backtest...
    //
    if (IO_Pointer != null)                        // Is there an I/O pointer from a prior backtest?
    {                                              // Yes.  Close it.
        WriteFile( IO_Pointer, "" );
        WriteFile( IO_Pointer, "==================================================== "
                             + "End of Session"
                             + " ====================================================" );
        WriteFile( IO_Pointer, "" );
        IO_Pointer.Close();                        // Close the output file.
        IO_Pointer = null;                         // Forget we had an open file.
    }
    base.Dispose();                                // Tell Ninja to do its cleanup.
}
//=====================================================================================================================

protected override void OnBarUpdate()
{
    if (IO_Pointer == null)                        // Need to create file?
    {                                              // Yes...
        string out_file_name = NinjaTrader.Cbi.Core.UserDataDir                  // Use folder: "MyDocuments\NinjaTrader 6\"
                             + "trace" + DateTime.Now.Ticks.ToString() + ".txt"; // Filename:   "trace<randomenumber>.txt"
        IO_Pointer = Create_File( out_file_name ); // Create the output file.
        WriteFile( IO_Pointer, "==================================================== "
                             + "Start of Session"
                             + " ==================================================" );
        Write_File( IO_Pointer, String.Format( "Strategy {0} has been initialized for instrument {1} [value: {2:c}/point or {3:c}/Tick] at {4:G}",
                                               base.Name, Instrument.FullName, Instrument.MasterInstrument.PointValue,
                                               (Instrument.MasterInstrument.PointValue*TickSize), DateTime.Now.TimeOfDay ) );
    }
}

Note this has the code to close the output file in the Dispose() method, so it will get done every time a backtest completes.

[Pete S]

1. For the time being I am doing this in my Initialize routine as a workaround:

if (IsBacktest != 0)
{
Random rand = new Random();
if (rand.Next() % 5000 == 0)
{
GC.Collect();
}
}

and it appears to be keeping 250M-300M of memory free. I'm not a .net programmer so this may be coincidental.

2. The save exceptions do occur without OutOfMemory exceptions. What I am doing is running an optimization against a folder/symbol list of 10 symbols, the optimization is on 60 minute data, the strategy takes around 7 parameters, each of which have 3 to 10 possible values.

The run takes approximately 2 hours. When I complete, and attempt to save the results, I get this stack trace:


************** Exception Text **************
Wilson.ORMapper.ORMapperException: ObjectSpace: Entity Object not being Tracked - NinjaTrader.Cbi.Instrument:fde20e1e3aa342dbb592026 4d9ca0d2e
at Wilson.ORMapper.Internals.Context.get_Item(Object entityObject)
at Wilson.ORMapper.Internals.Instance.PersistChildren (Transaction transaction, PersistDepth persistDepth, Boolean parentDeleted)
at Wilson.ORMapper.Internals.Instance.PersistChanges( Transaction transaction, PersistDepth persistDepth, Boolean parentDeleted)
at Wilson.ORMapper.Transaction.PersistChanges(ICollec tion entityObjects, PersistDepth persistDepth)
at Wilson.ORMapper.ObjectSpace.PersistChanges(ICollec tion entityObjects, PersistDepth persistDepth)
at Wilson.ORMapper.ObjectSpace.PersistChanges(Object entityObject, PersistDepth persistDepth)
at NinjaTrader.Strategy.SaveStrategyForm.OnOK(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventAr gs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

************** Loaded Assemblies **************
...

[KBJ]

I've found that using the optimizer efficiently takes a real knack.

When I run it, I usually try to optimize one parameter at a time. When done optimizing one parameter, I set that parameter setting to just use one or at most two values when going on to optimize another parameter.

Otherwise I find that there are just too many combinations, and it takes forever to run.

In your case, Pete, it runs out of memory. In my case it takes an excessive amount of time.

In either case, you can work around this by optimizing one parameter at a time, and using the optimized value when optimizing subsequent parameters.

This takes more hands-on, but you'll probably find that you can get your two-hour optimization done in a small fraction of the time -- like maybe 10-15 minutes -- but you'll have to stay there and tweak it.

The plus is that you should be able to do this without running out of memory.

Good luck.

[NinjaTrader_Dierk]

Pete,

1) We'll look into.
- In case you use statics or system ressources: make sure you clear them up in the Dispose method (please see the docs).
- Would you get your memory problem also on the simple MACrossOverSample provided, or is it only iwht one or more of your strategies?

2) This problem will be fixed with NT6.5. Thanks for reporting