Backtester slower and more memory than normal

[krugman25]

I have an issue where running a backtest is taking a lot more time and memory than it should. When I run a single test using "standard" it takes <5 seconds and doesn't move my RAM usage at all. When I take the same strategy and run it under optimization, each iteration takes over a minute and after 30 iterations my RAM usage is around 60GB(I have a 64GB machine). I am running the latest NT version12. I removed my work space and started fresh, which didn't do anything. I set IsInstantiatedOnEachOptimizationIteration to false which didn't do anything. I tried a lot of other little changes which didn't do anything. At this point I am at a loss.

UPDATE:
There is certainly something wrong with the NT8 backtester engine. I bit the bullet and took the same NT 8 code and ported it over to NT 7. I have done all of this with the newest NT7 64bit software. With the same everything: tick data, source code, settings, etc I was able to run the exact same test in less than a second that was taking 5 seconds in NT8. I than ran the same optimization and it is going about 1000 times faster, literally. Only took a couple seconds in NT7, was going to take 30-minutes to an hour in NT8 for only 80 iterations. Now onto memory usage. In NT8 my RAM usage was around 60GB after only 30 iterations of my optimization. I am currently running a much larger optimization in NT7(around 100k iterations), it is about halfway done and has not budge from 7GB since the test started. Not to mention 100k iterations in NT7 only takes around 10 minutes, 80 iterations in NT8 was 30-60 minutes. That's night and day difference.

I hope I can help NT resolve the issue.

[NinjaTrader_PatrickH]

Hello krugman25,

Thank you for your report.

Can you provide us your strategy that you used in NinjaTrader 8?
Do you see the same behavior with the SampleMaCrossover using the same instrument, from and to dates, as well as the same number of iterations through the parameters?

[krugman25]

Hello krugman25,

Thank you for your report.

Can you provide us your strategy that you used in NinjaTrader 8?
Do you see the same behavior with the SampleMaCrossover using the same instrument, from and to dates, as well as the same number of iterations through the parameters?

Yes I can reproduce the issue with the sample MA. I set it up to do 40,000 iterations, using 1 minute bars and 2 years of history(2014 to 2016). On NT 8 it said it was going to take 6 days, and my RAM was increasing by about 1GB per iteration(extremely slow iterations, 1 per minute). On NT 7 it took less than 15 minutes to do all 40k iterations and my RAM usage didn't hardly change at all. It was averaging about 1000 iterations per minute vs. 1 per minute on NT8.

I plan on monetizing the custom strategy/indicator I am working on so I would rather not send it, unless I absolutely need to. Since the SampleMA produces the exact same problems and behavior perhaps we can use that for now to resolve.

[NinjaTrader_PatrickH]

Hello krugman25,

Thank you for your patience.

May we set up a remote session to look into this on your installation?
If so, please send us a note to platformsupport[at]ninjatrader[dot]com with 'ATTN: Patrick H - 1557530' in the line. NinjaTrader Support is available from 8:30 am to 6 pm EST, please let us know in your note a time that we may reach you and include your telephone number.

I look forward to your response.

[krugman25]

Hello krugman25,

Thank you for your patience.

May we set up a remote session to look into this on your installation?
If so, please send us a note to platformsupport[at]ninjatrader[dot]com with 'ATTN: Patrick H - 1557530' in the line. NinjaTrader Support is available from 8:30 am to 6 pm EST, please let us know in your note a time that we may reach you and include your telephone number.

I look forward to your response.

I would like to do what I can to help NT out with this issue. Those are also my regular work hours so I will have to find a time that works best for me and then email you with that time. I have some vacation time coming up next month where I should be available on a weekday. I will let you know.

[krugman25]

I've been strictly using NT7 for the past week or two since I have been way too busy to look into the NT8 issues anymore. Today I was able to fire up NT8 and re-ran the optimizer on the MASampleCrossover strategy on both NT7 and NT8 just to do another comparison. This time the speed of both were very comparable. I am not sure what I may have done to change anything. I have made changes here and there like re-importing my tick data and some other OS level changes. I will be watching for the problem to resurface but for the time being it seems fine. But the memory issue is still in full force. For the test I ran 20,000k iterations of the MASampleCrossver on both NT7 and NT8. On NT7 it initially increases my memory usage by about 2Gbs at the very beginning (I am guessing it is caching the data series?) and then pretty much stays at that amount for the entire test. With NT8 I initially get a very large spike in memory usage and then it slowly grows. I let the MA crossover do about 5000 iterations(25% of the total test), and NT8 is using up 21Gbs of memory (4Gbs for the OS and other applications). That is a major difference compared to NT7 which may use 2-4Gbs at the most. I'm not saving a huge # of results so I know it's not from that. From the screenshot you can see I am only keeping 10 results. There is another issue I see which is after the test is finished it never releases the memory. I aborted the test early just to see what the memory usage would drop to and it stayed at 25Gb. After closing NT8, my memory usage drops to 4GB, which means NT8 was holding onto 21Gbs worth of memory. I was reading around online and saw a few people mentioned huge log files. Has that ever been proven or is it something worth looking into? I can't for the life of me imagine why NT8 is using 21Gbs for very simple moving average strategy, and why it continues to hold onto all of that memory even after the testing is over.

Note: Screenshot as attachment

[krugman25]

UPDATE: I stand corrected, I ran the exact same MA Crossover test on NT7, and that also used up 60GB of memory. So that begs the question, why does 1GB of tick data become 60Gbs of memory usage when running the optimizer? Is there anything I can do to improve on that, other than shorted my timeframe. I'm a firm believer in back testing on bid/ask tick data and running my EAs on timeframes no shorter than 10 years. I though that 64 gigs of ram would be more than enough. There is still an issue that is unique to NT8 which is the optimizer doesn't release the memory used after the testing is finished, whereas NT7 does release it all after it is finished.

So I spent a few hours of my Saturday running a gamut of tests on NT8 to see if I could pinpoint a setting or something that resolves the huge memory usage issue. After doing all of my testing I have come to the conclusion it is most likely a problem under the hood, in the source code of the engine, that I will not be able to pinpoint through just changing settings. I ran 3 tests and they all had the same behavior, which is the ram filled up to 60+Gbs before the optimization actually started. The period where the RAM fills up is between the point where you click the button the start the optimization and the point where it actually starts iterating through the test. That's the part where I believe the optimizer is loading up the data series and doing whatever else it needs to do before the testing starts.

All tests were performed on the built-in SampleMACrossover strategy. All tests were ran on 2 years of tick data(2014-2016). The tests I did are as follows:
1. Run test with MaximumLookback set to infinite
2. Run test with MaximumLookback set to 256
3. Disable all CPU's except for 1 and run test (I thought perhaps NT8 was trying to load the data series for each CPU thus loading 8 or 16 copies of the data). This proved to not be the case, even with 1 CPU the RAM still climbed up to 60+Gbs before the testing ever was able to start.

I'm out of ideas at this point.

I've attached screenshots of each test.

[krugman25]

Update #2: I ran some more testing and it seems to me the really big problem is just how much memory is being consumed when optimizing. Yesterday I wrote an application that took all of my tick data and got rid of duplicate ticks. This shrunk my original tick file by 40%. Then in NT8 optimizer I selected SampleMACrossover and selected 1 tick data series for 1 year of time(2015-2016). I then set it up so there would only be 2 iterations. When I ran the optimizer it consumed 8 GBs of memory. So roughly 4 GBs per iteration. I ran it again and this time made 3 iterations and it consumed almost 12 GBs, again about 4 GBs per iteration.
Based on these short and simple tests I would have to assume that the optimizer is queuing up multiple tests. My first question is how many of these tests is it simultaneously queuing up. Is it 4 because I have 4 cores, or 8 because I have 8 threads, or some other arbitrary number. The second question is exactly what is the optimizer doing to be consuming 4 GBs per iteration? That 1 year of tick data is roughly 500MB, so even if it was queuing up that same set of tick data for each iteration, that is no where near the 4Gbs per iteration that I am seeing. I need Ninja Trader to please explain what is happening behind the scenes in the back testing engine that would explain such huge amounts of RAM usage.

Update #3: I did try another test which was rather than use tick data just use minute bars. With that test I was still unable to optimize with NT8. The memory climbed back up to 60+Gbs and the optimizer never began running through the test. With NT7 this made a huge difference. Memory jumped by about 1Gb, and then immediately began testing. It ran through all 125 iterations in about 1 minute. There are some clear memory issues that I am seeing with NT8 vs NT7, especially with minute bars. The results from NT7 makes sense because 2 years of minute bars is simple not that large in size, perhaps a few MBs

[NinjaTrader_PatrickH]

Hello krugman25,

Thank you for your patience.

Running large spans of time (such as two years) on intra-day bar types can take some time to complete and when we increase this with the range (Min and Max) of the parameters when optimizing, memory usage will increase.

You would consider decreasing the span of time (Start and End date), the number of iterations (Min and Max range), and even disconnect from the data provider while running the tests to avoid downloading data each time you run the test.

Just to clarify on this matter; you were seeing increases in ram usage as well as CPU when using large date ranges and parameter ranges resulting in a large number of iterations. Ram will increase for the executions taken in each test. You were seeing that the performance is similar in NinjaTrader 7 as well after testing. Please let me know if this is incorrect or if you have any questions.

[krugman25]

Hello krugman25,

Thank you for your patience.

Running large spans of time (such as two years) on intra-day bar types can take some time to complete and when we increase this with the range (Min and Max) of the parameters when optimizing, memory usage will increase.

You would consider decreasing the span of time (Start and End date), the number of iterations (Min and Max range), and even disconnect from the data provider while running the tests to avoid downloading data each time you run the test.

Just to clarify on this matter; you were seeing increases in ram usage as well as CPU when using large date ranges and parameter ranges resulting in a large number of iterations. Ram will increase for the executions taken in each test. You were seeing that the performance is similar in NinjaTrader 7 as well after testing. Please let me know if this is incorrect or if you have any questions.

Yes I agree in general with your statements but I believe that, all things being equal, NT8 uses significantly more RAM for the exact same tests. I will try and get some good screenshots to show what I mean. I will attach once I have them.

Also I didn't see you make mention of the issue where NT8 doesn't release the RAM after testing, which seems to be something unique I have seen with NT8.

[krugman25]

Screenshots may speak better to the issue than trying to explain it.

Here are the results of testing I just did. The settings used for the tests and the tick data are identical in both NT7 and NT8. The following are the settings used:

Strategy:Sample MA Crossover
Fast: 10;100;1
Slow: 25;250;1
Instrument: EURUSD
Price: Tick
Value: 1
Start Date: 08/01/2015
End Date: 08/01/2016
Maximum Bars Lookback: 256

In the attached screenshots you will see the results as follows
NT7
* Test begins: Memory usage at 4.8GB
* 1 minute after test began: Memory Usage around 19GB
* 6 minutes after test began: Memory is holding steady between 16-25MB. Every few minutes will climb to 25MB, and then drop to 16MB and repeat.

NT8
* Test begins: Memory usage at 5.4GB
* 1 minute after test began: Memory Usage around 14GB
* 6 minutes after test began: Memory is holding steady at 50GBs.

The results shows that all things being equal, in this specific case that NT8 uses double the memory. I want to be clear here on what my issue is. The primary issue is that because NT8 uses a lot more memory, I have to shorten my time frame quite a bit compared to NT7. For example, in NT7 I have no problem running a 2 year tick data back test, but that same test on NT8 is impossible since it runs out of RAM before it is able to start. Just for the record a 2 year tick backtest in NT7 uses about 45Gbs of RAM. According to my testing it looks like on average that NT8 uses twice as much memory than NT7.

Note: Due to attachment limitations, I have attached the NT7 screenshots to this post. There is a starting, 60 seconds after start and 6 minutes after start screenshots.

[krugman25]

Due to attachment limitations, I have attached the NT8 screenshots to this post. There is a starting, 60 seconds after start and 6 minutes after start screenshots.

[krugman25]

I know NT developers are very busy. I didn't see any comments on this issue from the new screenshots and information I provided last night. Are there any new ideas on this or tests I can run to help NT developers pinpoint the problem?

[NinjaTrader_PatrickH]

krugman25,

Thank you for your patience.

I am reviewing your case information and I will follow up when I have further details.

[NinjaTrader_PatrickH]

Hello krugman25,

Thank you for your patience.

The memory would not be released by NinjaTrader. It is marked by NinjaTrader when it is done with that memory that it no longer needs it. Then the .NET garbage collection would need to run through to clear out the marked memory. There is no means to tell .Net when to run garbage collection.

I noticed that my memory was maxed out in both NinjaTrader 7 and 8 when running a year long or two year long 1 Tick bar based optimization with over 20,000 iterations to run.