Out of Memory Error thrown.

[forrestang]

Hello.

So I've created a script who's purpose is just to take data from charts, and print it to the output window so I can copy and past in excel. I have no intention of actually using the script live.

Admittedly it is somewhat inefficient, has many loops and loops back over previous bars almost every bar to get various values. It is also has 3 MTF bar series added to it. The primary data series is a 1 minute bar.

Running this on a few years is no problem. But I want to get data for about 10 years all in one go. The reason being that some of the data will be sequenced, and I want the sequence number of one data point to match another. If I split it up I will either have to ditch the sequencing or go in and manually adjust it which will take a lot of time.

I've tried obvious things like restarted my PC, and shutting down any back ground apps that I don't need running on Windows X.

So I'm just wondering if there are any solutions that might give my script a better shot at completing? If not, I will either just work on outputting my data in several pieces and painstakingly correcting them, or take the time to rewrite my script so that it is a bit more efficient.

Thanks for the help.

[NinjaTrader_Brandon]

Hello forrestang,
Thanks for your post.
Are you using the 32 bit or 64 bit version?
How much memory is in your computer?

[sledge]

You could write a formula into excel and paste it into the column and increment by 1.

[forrestang]

Hello forrestang,
Thanks for your post.
Are you using the 32 bit or 64 bit version?
How much memory is in your computer?

I should have posted my PC specs, will attach them below. But I initially just tried using the 32 bit version which is all I ever use. NT7 would throw the memory error after about 30 seconds.

After my post, I tried it in the 64 bit version, and this time it ran for quite some time before my PC shutdown. I walked away so I didn't see why it shut down. I will try it again though with the 64 bit version.

In task manager, the memory usage was going as high as 96%.

PC Specs:
Intel i7 930, 2.8GHz oc'd to 3.6Ghz
EVGA X58 SLI -mobo
EVGA 780 Classified (4GB)
Prolimatech Megahelm Cooler
6GB Patriot PC-12800
Corsair TX650 watt PSU
240GB Intel 330 SSD, Western Digital Green 500GB sata, 64mb cache
Windows 10

[forrestang]

You could write a formula into excel and paste it into the column and increment by 1.

Thanks for reply, that is likely what I will wind up doing in excel or matlab if I can't export it all in one go.

[NinjaTrader_Brandon]

Hello forrestang,
Thanks for your reply.
You could give that a try or upgrading the amount of memory in your system.

[forrestang]

I ran this script in "Safe Mode" with virtually nothing else running which seemed to help out a lot. It finally completed my stats print out but it took quit a while.

Also the 64bit version seems to handle running so low on memory a ton, where the 32 bit version would just eventually crash. This isnt' a typical script, and wouldn't run so much data on my typical indies.

I wonder if it was just the loops I had running, or if the additional TFs added along with all the data is what really bogs it down? As I've exported 10 years of data via script with no real problems. But this one just took forever to finish.

[sledge]

Hello.

So I've created a script who's purpose is just to take data from charts, and print it to the output window so I can copy and past in excel. I have no intention of actually using the script live.

Admittedly it is somewhat inefficient, has many loops and loops back over previous bars almost every bar to get various values. It is also has 3 MTF bar series added to it. The primary data series is a 1 minute bar.

Running this on a few years is no problem. But I want to get data for about 10 years all in one go. The reason being that some of the data will be sequenced, and I want the sequence number of one data point to match another. If I split it up I will either have to ditch the sequencing or go in and manually adjust it which will take a lot of time.

I have no idea how much data you are trying to cut and paste or what format it is in....

You might want to increase your system RAM to 32gb+... or change your code to NOT output to the OUTPUT window but to a file...

[forrestang]

I have no idea how much data you are trying to cut and paste or what format it is in....

You might want to increase your system RAM to 32gb+... or change your code to NOT output to the OUTPUT window but to a file...

Is writing to a file less memory intensive than printing to the output window?

[sledge]

Is writing to a file less memory intensive than printing to the output window?

Well, you it wouldn't be using RAM memory... just disk space... this process of yours will be slower to complete.

[rayko]

You could use StreamWriter to write to disk.

Yes, it would be a bit slower but it won`t crash on memory and you would not have to copy/paste it into Excel if you write the file with a .txt suffix and separate variables out using a ASCI Tab "\t" like this:

using(StreamWriter writetext = new StreamWriter("write.txt"))
{
string strTextout = VarA.ToString() + "\t" + VarB.ToString();
writetext.WriteLine(strTextout );
}

The asci tabs will make Excel separate the variable output into columns while its being opened into Excel.

Here is a an example from StackOverflow:

http://stackoverflow.com/questions/7569904/easiest-way-to-read-from-and-write-to-files

[forrestang]

So I got my Matlab script to write the desired NT script so that I can then paste that script into the onBarUpdate block.

It also works exactly the way that I want it to. I am able to backtest ideas that I can prototype very quickly in Matlab, as doing the same thing in C# in NT usually takes me days to accomplish what I can do in Matlab in minutes.

Unfortunately, the way I am accomplishing this is to simply have a loop in Matlab produce the desired entry points in time and price, and pasting that data in the NT script. That way I can leverage the backtesting engine of NT, but do my analysis elsewhere which is easy to do.

I will attach a sample piece of code of the output I am pasting into the NT editor below. Each instance has a different entry point, and time associated with it. So this means that there are MANY, MANY lines

So my question is, is there a limit to the number of lines that NT can process within a script? For example, I've pasted up to 20K lines in with no problem and it backtests just fine. I tried data for 30K lines in the editor and NT(64bit) crashes. It doesn't throw a memory exception or anything, it just crashes. Also my memory/CPU usage isn't going to a dangerous amount that would seem to be a problem.

The solution as I see it is just to do several chunks at a time of the data which is feasible, but I am just wondering if there are any known editor limits?

This isn't the finished output, as now I'm just testing to make sure it works properly as there will be sizing and other criteria. But the basic produced NT script I am pasting is of the form:

#region code
DateTime sessStart1511= new DateTime(2016,03,06,16,05,00 ); 
DateTime sessEnd1511= new DateTime(2016,03,07,16,00,00 ); 
if( Time[0]>=sessStart1511 &&Time[0]<=sessEnd1511 )  
{  
    if(Time[0] ==sessStart1511)   { //Initialize OpenBar 
        DrawVerticalLine("testO"+CurrentBar,0,Color.Green,DashStyle.Dot,1); 
        openTF1=Open[0];  tradeCnt=0; 
    }  

    if(High[0]> openTF1+25*TickSize &&tradeCnt==0)   {  
        DrawVerticalLine("Ent"+CurrentBar,0,Color.Blue,DashStyle.Dot,1); 
        tradeCnt++; 
        EnterLong(10000,"myLong"); 
     }  

    if(Time[0] ==sessEnd1511 )   { //Report Close of Bar 
        DrawVerticalLine("EOD"+CurrentBar,0,Color.Red,DashStyle.Dot,1); 
        ExitLong("myLong"); 
    }  
}  
.
.
.
.
...... Many more of the above code just with different criteria for time and price, but same format
#endregion