Backtester slower and more memory than normal

[krugman25]

Hello krugman25,

Thank you for your patience.

The memory would not be released by NinjaTrader. It is marked by NinjaTrader when it is done with that memory that it no longer needs it. Then the .NET garbage collection would need to run through to clear out the marked memory. There is no means to tell .Net when to run garbage collection.

I noticed that my memory was maxed out in both NinjaTrader 7 and 8 when running a year long or two year long 1 Tick bar based optimization with over 20,000 iterations to run.

About the memory not being release, it seems to me when NT7 is finished optimizing the memory usage drops down to what it was before optimizing starts. In NT8 is doesn't. There are other threads floating around the interwebs making this same claim so I know I'm not alone.


In regards to the memory usage...
Soooo did you try a smaller time frame until your machine could handle it? From your post I read between the lines that your machine couldn't handle a year of tick data, the end? I apologize if this isn't case and I am missing something, but if that is the case then that was a really lazy attempt to resolve the issue. Clearly if your test machine couldn't handle 1 or 2 years of tick data, then how about dropping it until it can? If you maxed out memory in both NT7 and NT8 then you didn't really test out the issue. If my set my tick data to 6 years, I would too max out my memory in both NT7 and NT8 and it wouldn't tell me anything about the discrepancy between the two.

I know I'm being forward but I am a technical manager of a very competent group of software developers, and the responses I am seeing here are confusing and disheartening.

I never claimed that there was a bug using exactly 1 or 2 years of tick data. That just happened to be able how much data my computer can handle. If yours can't handle that much then drop the time frame until it can. If the issue is that tick data consumes twice as much memory in NT8 vs. NT7, which is exactly what I am claiming, then that should be noticeable true for any time frame. No?

[NinjaTrader_PatrickH]

Hello krugman25,

Thank you for your response.

The PC I tested only has 8 GBs of ram so it did not take long for such a large set of variables to generate a high ram usage.

I tested with smaller time frames and with smaller ranges for the Min and Max of my parameters and I see NinjaTrader 8 moving very quickly and memory usage not growing or leaking. In comparison to NinjaTrader 7, the NinjaTrader 8 memory usage was well below that of NinjaTrader 7.

Try disconnecting from your provider after ensuring you have enough data. Then run your tests.

[krugman25]

Hello krugman25,

Thank you for your response.

The PC I tested only has 8 GBs of ram so it did not take long for such a large set of variables to generate a high ram usage.

I tested with smaller time frames and with smaller ranges for the Min and Max of my parameters and I see NinjaTrader 8 moving very quickly and memory usage not growing or leaking. In comparison to NinjaTrader 7, the NinjaTrader 8 memory usage was well below that of NinjaTrader 7.

Try disconnecting from your provider after ensuring you have enough data. Then run your tests.

I don't use any connections for my back testing, although it does by default connect to TD Ameritrade. As mentioned in my previous posts I import a tick csv from dukascopy. My dukascopy tick data file has been shrunk by about 40% by eliminating duplicate ticks. The tick csv file size for 5 years of tick data is 2GB. These tests I am doing are on 1 year of tick data, which would be roughly < 500Mb.

I did disable my TD Ameritrade connection as you requested. I also deleted and re-imported the Dukascopy tick data and then re-ran my test. Please see my screenshots for testing results.

Results as follows:

NT7

• At the 5 minute mark the memory usage was steady at 15GB.
• The memory reached 15GB within the first 2 minutes and then stayed at that usage for the rest of the testing
  
• I let the optimization run for a while longer and the memory usage continued to stay at 15GB

NT8

• At the 6 minute mark the memory usage was around 46GB
• It continued to rise throughout the entire 6 minutes.
• I am not sure if it would have continued to rise because I killed the test, but the pattern I saw is that it would rise, then steady out for a few seconds and then rise a little more.
  

I agree the speed is about the same. That much can be seen in the screenshots. Both were going to take around 10 hours to finish. But there is a clear difference between the 2 softwares in terms of memory usage. In this testing the computer is the same, the tick data is the same, the optimization settings are the same, and the tests themselves were performed in quick succession one immediately after the other. The only difference that could be is the software engine themselves. I am not saying there is necessarily a bug but I'm going to try and think outside of the box on this. Is it possible that NT8 is launching more threads or has greater degrees of parallelism when running multiple simultaneous back tests? Or are the collections that are used to store the tick data for back testing different? In all of my years of being a C# developer I have only written a memory leak once. It's actually extremely difficult to do in C#, thankfully, but perhaps there is a bug somewhere else? Local copies of containers, or threads not dying, or something else? Are there any useful logs or NT database information being outputted by NT8 that can help see whats going on internally. Is there any settings file or database setting where I can adjust how many simultaneous optimization threads(degrees of parallelism) are allowed to run at one time?

I am very close to just going ahead and writing in a whole routine that outputs to an external source a bunch of diagnostics information during optimization so I can get a much better picture of whats going on in the engine between NT7 and 8.

[krugman25]

Update # 2: A Second Test

I ran a completely different type of test this evening and came up with some very interesting and what I believe are useful results. The test was the same Sample MA Crossover test I have been running but with 1 month of tick data using NT8 32-bit and NT8 64-bit. I could only go as high as 1 month of tick data because any more would cause the 32-bit version to crash, which I am guessing is due to the memory limitations of 32 bit applications. The result was that the 64-bit version was using as much as 4 times more memory than the 32 bit version, both are NT8.

Here are my notes:
64bit

• 64bit at the 3 minute mark had grown by 4GBs(was 4GBs before test start, grew to 8.2GBs within 3 minutes.
• 64bit continued to grow for the first 2 1/2 minutes and steadied out at the 3 minute mark

32bit

• 32bit at the 3 minute mark had grown by 1GB(was around 4GBs before test, grew to between 5-5.2GBs
• 32bit memory usage peaked within 30 seconds and stayed steady at that mark for the next few minutes. I noticed that the memory usage peaked quicker than the 64bit which continued to slowly grow for a few minutes.
• 32bit behaved much like the NT7 where memory would peak quickly and then would oscillate around that value a little bit as it ran through tests. In comparison the 64bit continued to grow in memory for a much longer period of time and ultimately when it steadied out it would consume more memory.

I wish I could run this test on a larger set of data so the contrast would be larger but it doesn't really matter what time frame is used. The principle remains the same. Which is that in general I am seeing that the 64bit NT8 will use 2 - 4 times more memory for the exact same tests compared to NT7 and now NT8 32 bit.

[krugman25]

Bump...Bump

[NinjaTrader_PatrickH]

Hello krugman25,

Thank you for your patience.

Apologizes for the delay in getting back to you. I am looking into this on my end and I will follow up with you when I have more details.

[krugman25]

Hello krugman25,

Thank you for your patience.

Apologizes for the delay in getting back to you. I am looking into this on my end and I will follow up with you when I have more details.

Great! Thanks for the update Patrick.

[NinjaTrader_PatrickH]

Hello krugman25,

Thank you for your patience.

I would recommend taking a look at the information at the following link on optimizations, historical data, memory usage, and 32 bit versus 64 bit: http://ninjatrader.com/support/helpG...ionPerformance

However, the crash of 32 bit is unexpected. Is this reproducible? I was unable to reproduce the same crash on my end in 32 bit.

[NinjaTrader_Matthew]

Just to let you know, we've been reviewing this post along with Patrick, but do not see anything out of the ordinary with our testing in NT8. We believe what you're seeing here is the result of the improvements we made in multi-threading. In reviewing your results, I would expect it to use quite a bit more memory because of the way results are stored in memory.

NT7 would keep only the current # best results in memory. NT8, we'll keep X more permutations in memory, where X = the number of logical processors on your system * Keep best results settings.

For example, where NT7 might have only kept 10 results in memory - NT8, on a machine with 8 cores, it will keep up to 80 results.

If the amount of memory you see utilized is a problem, aside from reducing the number of days to load, you can try reducing the number of "keep best results," which should significantly lower the number of permutations held in memory.

Depending on the criteria of your backtest, you can opt to set IncudeTradeHistory to false:

http://ninjatrader.com/support/helpGuides/nt8/en-us/includetradehistoryinbacktest.htm

This option is set to true by default in the SampleMACrossOver you are testing. If you make a copy and set that property to false, you will lose detailed trade information, but should significantly impact the amount of memory used per backtest.

[krugman25]

Just to let you know, we've been reviewing this post along with Patrick, but do not see anything out of the ordinary with our testing in NT8. We believe what you're seeing here is the result of the improvements we made in multi-threading. In reviewing your results, I would expect it to use quite a bit more memory because of the way results are stored in memory.

NT7 would keep only the current # best results in memory. NT8, we'll keep X more permutations in memory, where X = the number of logical processors on your system * Keep best results settings.

For example, where NT7 might have only kept 10 results in memory - NT8, on a machine with 8 cores, it will keep up to 80 results.

If the amount of memory you see utilized is a problem, aside from reducing the number of days to load, you can try reducing the number of "keep best results," which should significantly lower the number of permutations held in memory.

Depending on the criteria of your backtest, you can opt to set IncudeTradeHistory to false:

http://ninjatrader.com/support/helpGuides/nt8/en-us/includetradehistoryinbacktest.htm

This option is set to true by default in the SampleMACrossOver you are testing. If you make a copy and set that property to false, you will lose detailed trade information, but should significantly impact the amount of memory used per backtest.

Thank you for your response Matthew. I will test it with that property set to false as you mentioned in your post. I also have a 32Core/64GB data processing server that I am considering running a test on just to see how it performs on different hardware. Although if what you say is true then it will keep x32 of the results which may even be worse on the memory.

I noticed you mentioned NT7 vs. NT8 in your post. Is the principle the same for 32bit vs 64bit? I discussed some testing in a previous post where I compared NT8 32bit vs NT8 64bit, and I had the same results as if I did an NT7 vs NT8 64bit test, where the 64bit version was using 4 times more memory than the 32bit version. In your explanation you discussed NT7 vs NT8, but I would be curious if that also applies to 32bit vs 64bit. Otherwise if not how would you explain the memory usage discrepancy between the 32 and 64 bit versions of NT8?

[NinjaTrader_Matthew]

At the present time, during an Optimization, IncludeTradeHistoryInBacktest is forced to false in a 32-bit process. This policy was implemented to conserve memory usage in systems which may not be able to accommodate larger tests (although the policy is subject to change during beta based on client feedback)

If you manually set this option to false, you should see similar results as far as memory goes during optimization between the 32-bit and 64-bit processed. But please keep us in the loop on your testing.

[krugman25]

Just to let you know, we've been reviewing this post along with Patrick, but do not see anything out of the ordinary with our testing in NT8. We believe what you're seeing here is the result of the improvements we made in multi-threading. In reviewing your results, I would expect it to use quite a bit more memory because of the way results are stored in memory.

NT7 would keep only the current # best results in memory. NT8, we'll keep X more permutations in memory, where X = the number of logical processors on your system * Keep best results settings.

For example, where NT7 might have only kept 10 results in memory - NT8, on a machine with 8 cores, it will keep up to 80 results.

If the amount of memory you see utilized is a problem, aside from reducing the number of days to load, you can try reducing the number of "keep best results," which should significantly lower the number of permutations held in memory.

Depending on the criteria of your backtest, you can opt to set IncudeTradeHistory to false:

http://ninjatrader.com/support/helpGuides/nt8/en-us/includetradehistoryinbacktest.htm

This option is set to true by default in the SampleMACrossOver you are testing. If you make a copy and set that property to false, you will lose detailed trade information, but should significantly impact the amount of memory used per backtest.

I set " IncludeTradeHistoryInBacktest" to false and it didn't seem to make much difference. I see what you are saying in regards to how much memory each results uses. NT8 certain uses more memory than NT7 in that regards, like a serious amount more memory. In NT7 I can run 6 years of tick by tick testing and keep tens of thousands of results no problem, NT8 uses that same amount of space if I run 1 year of tick back testing and keep 10 results. Heck in MT4 and MT5 I used to run 10 years of tick back testing and keep 20,000+ results without any problem. That means each NT8 backtest results are 100's of times bulkier than a result on just about any other software out there. Since IncludeTradeHistoryInBacktest didn't make much of a difference what else are those tests storing that NT7 is not, that could possibly be consuming 30-40 Gigabytes for just a dozen tests? Is there any way now or plans in the future to allow users to change how many permutations in memory are allowed, how many cores are used, or even cut more stuff out of the back test results to get them similar to NT7's size? That's all I am looking for at this point because I can run 15 year tick by tick tests on NT7 all day long no problem.

Doing tests on minute or hour bars, or on a few months of data doesn't work for power users and serious quant developers. Anything less and the results are moot, imho. I do consider NT8's backtesting engine to be superior, so it's disappointing that it will basically be unusable for me until I can get a computer with about 1000GB of RAM.

[krugman25]

At the present time, during an Optimization, IncludeTradeHistoryInBacktest is forced to false in a 32-bit process. This policy was implemented to conserve memory usage in systems which may not be able to accommodate larger tests (although the policy is subject to change during beta based on client feedback)

If you manually set this option to false, you should see similar results as far as memory goes during optimization between the 32-bit and 64-bit processed. But please keep us in the loop on your testing.

Thank you for the quick response Matthew. I stand corrected on a comment I made earlier about "IncludeTradeHistoryInBacktest" not making a difference. I just finished up some testing and it did make a little bit of a difference.

I wanted to just run another gamit of testing now that I have some explanation from NT why the memory usage might be so high. This testing again showed major memory usage differences between NT8 64bit, and NT7/NT8 32bit. The settings and data I used was identical in all testing. I used 1 tick data for a 1 month time frame(7/1/2016 to 8/1/2016). I had it run generic optimizing and save 100 results. I used the MACrossover strategy that's included with NT software. I waited until the memory usage had completely leveled out and wasn't growing anymore before I recorded the usage. The following are the tests and results

NT8 x64 (IncludeTradeHistoryInBacktest = True) - 24GB used
NT8 x64 (IncludeTradeHistoryInBacktest = False) - 17.5GB used
NT8 x32 (IncludeTradeHistoryInBacktest = False) - 0.7GB used
NT7 x64 - 1GB used

As you can see from the tests NT8 x64, even with "IncludeTradeHistoryInBacktest = False" it still used 25-30x more memory compared to NT8 x32, and 20-25x more memory when compared to NT7 x64.

Not even looking at NT7, I understand NT8 is a superior product with more back test result information for the benefit of the back tester, but 30x more memory used just because the software is 64bit doesn't seem right. I would hope there would be a way to bring that way down in par with memory usage in the 32bit version of NT8.

[krugman25]

krugman25,

Thank you for your patience.

I am reviewing your case information and I will follow up when I have further details.

I must have missed this post yesterday. I read through the article and while very informative it focuses mainly on the trade history property. Having IncludeTradeHistoryInBacktest set to true, the x64 program was using 35 times more memory than the x32 version. Setting "IncludeTradeHistoryInBacktest" to false did help save 7.5GB, but even then it was still using a whopping 25 times more memory than that same test on x32. There must be something other than trade history that is being stored into RAM to account for that 25x larger memory usage. Matthew had mentioned that it may be storing x times # of permutations , x being = to # of cores. Even then my machine has 8 logical cores so I would only expect it to use, at most, 7x more memory.

Please refer to my previous post for testing results between x64 and x32 that I performed last night.

[NinjaTrader_Matthew]

Your NT7 results vs NT8 were definitely surprising, however, I am more surprised at how little memory NT7 is reporting.

If I use the same input setting you are using (1 month, EURUSD, 1 tick bid data), both NT7 and NT8 use up to 12GB of the 15GB total on my machine.



I'm not quite sure what the difference would be and why NT7 would only use 1GB in that scenario. However, since we're looking at Forex data, the tenth pip granularity in NT8 could come into play.

Since we are dealing with forex data: in NT7, From Tools-> Options-> Misc-> is the "Quote currencies in" set to Full Pip, or Tenth Pip?

If it is full pip, try changing it to Tenth Pip--> do you still only see about 1GB used?