Optimizer RAM and CPU usage

[MarvellousMoneyMachine]

The NT optimizer seems to be using system resources in a very strange way.

When I backtest my pairs-trading strategy with a single set of parameters, the backtest will take a reasonable 10 seconds and use up about 100MB worth of RAM. When I run an optimization that has to backtest 10 times (10 possible combinations of parameters), NT uses about 1GB of RAM and takes about 100 seconds.

A problem arises when I try to run optimizations that take 50 backtests or more. The optimization does not scale properly, it takes an hour instead of 500 seconds. NT always follows the same pattern regardless of which instruments I run the strategy on:

1. CPU usage immediately spikes to 70%
2. RAM usage slowly crawls up to about 11GB
3. When RAM usage hits 11GB, CPU usage decreases to about 15%

I assume that the CPU is used less because NT is now fetching data from a hard drive and this introduces a fair amount of latency into the operation.

What I want to know is if anyone else has had this problem and if there is some way to fix it. I have a feeling that I might be doing something wrong.

[NinjaTrader_Brett]

Hello,

Thanks for the note.

NT is multi threaded with its back tester so it will run optimize runs concurrently. If you have a bottleneck and your no longer able to run these optimize runs concurrently they have to queue up and run one after the other in a linear fashion. This is why you see the increased back test time once you queue up too many back tests over the amount your are able to run asynchronously.

This most likely is what is going on in your case.

-Brett

[MarvellousMoneyMachine]

Thank you for the speedy response!

What is the best way to deal with this problem? Should I run the optimizations in small batches?

[NinjaTrader_Brett]

I like to do it that way personally but not needed in the end the time it takes to run in small batches would equal the time to run it all at once.

Only reason I like to do it in small batches personally is if I get some error or PC locks up or power goes out or windows updates forces a shutdown etc. etc. I'm not out a bunch of testing data.

-Brett

[MarvellousMoneyMachine]

I think I found the problem. The C# 'print' function seems to be the thing which was using up all my RAM.

Removing it from my code has made the optimizer work properly.