NT becomes unresponsive once swapping starts with optimizer backtesting

[MojoJojo]

Hi,

while backtesting, I've run into the problem that NT becomes unresponsive and to me it appears the problem is that it no longer functions properly once swapping occurs.
In order to locate the issue I've tested the SampleMACrossOverStrategy with following parameters and nothing else running:
Fast: 10;50;1
Slow: 100;300;1
Period: 1 minute
Instrument: ES
Time period: YTD, ~5 months.
Iterations: ~8000

RAM: 16 GB
CPU: i7, 4 cores 8 threads

Case 1. Keep best result#: 500
No problem at all, no swapping, no lock ups

Case 2. Keep best result#: 1000
ok

Case 3. Keep best result#: 2000
starts to swap and maybe one micro stutter

Case 4. Keep best result#: 3000
lock ups start past the half way mark but completes in 8 minutes, best estimate was 2 minutes. The delay was due to the lock ups and not iterations taking longer

I've started with a fresh workspace before each test in order for garbage collection to occur in case it matters.
Once swapping happens NT becomes unusable despite CPU utilization dropping to about 1 core.
The swapping did not occur until 4000-5000 iterations, which makes me wonder if results exceeding 3000 were really disposed of, but then again it was fine with 500.

Maybe the root of the issue is not on NT's side and the obvious solution is to add more RAM and test fewer cases.

It wasn't obvious that RAM was the source of the problem but the documentation mentions limiting the result size.

https://ninjatrader.com/support/help...a_strategy.htm

[NinjaTrader_Jesse]

Hello MojoJojo,

Thank you for the post.

I just wanted to provide a quick note here and let you know that I will be testing these steps. If I can report anything here I will however as you noted this likely can be avoided by using more limited testing. When paging occurs in windows that will in general slow down performance and delay operations for a variety of reasons. That will likely include other programs and also parts of the platform. Because this is a situation where more memory is being requested the paging will continue to happen for as long as needed to see the operation through so that could impact performance for a considerable amount of time. This is not necessarily something that is specific to NinjaTrader but more so of any application which can utilize the ram fully or exceed the amount you have.


I look forward to being of further assistance.

[MojoJojo]

Hello Jesse,

thanks for investigating the issue, I appreciate it.

My intention is just for people to be aware that memory shortage can be a major source of performance problems.

I am pretty certain that NT's lock ups were not caused by internal logic, during previous runs even my graphics driver crashed so the system was just overloaded.
Nevertheless some optimization never hurts, despite it being often difficult and the pay off hardly worth it . But sometimes there are exceptions.

[MojoJojo]

For specifically limiting maximum memory usage when running the optimizer I've given manually invoking the garbage collector a shot.
While this comes with a minor performances hit and the garbage collector makes NT unresponsive for about a second or less each time it is invoked, it's not a terrible compromise for me.

    public class OptimizerManualGarbageCollection {
        private int iterations = 0;
        private static OptimizerManualGarbageCollection instance;

        private OptimizerManualGarbageCollection() {    
        }

        public static OptimizerManualGarbageCollection GetInstance() {
            if(instance == null) {
                instance = new OptimizerManualGarbageCollection();
            }
            return instance;
        }

        public void NewIteration(int minIterationsBeforeCollection) {
            if(minIterationsBeforeCollection <= iterations) {
                GC.Collect();
                GC.WaitForPendingFinalizers();
                iterations = 0;
            } else {
                iterations++;
            }
        }
    }

Strategy code

        private OptimizerManualGarbageCollection optimizerManualGarbageCollection = OptimizerManualGarbageCollection.GetInstance();
...
            } else if (State == State.Configure) {
                if(Category == Category.Optimize || Category == Category.MultiObjective || Category == Category.WalkForward) {
                    lock(optimizerManualGarbageCollection) {
                        optimizerManualGarbageCollection.NewIteration(GarbageCollectionInterval);
                    }
                }

The GarbageCollectionInterval should be chosen depending on how resource intensive iterations are and the ceiling for memory usage, I've chosen 50 for slow iterations.
Less frequent garbage collection leads to fewer interruptions but in my experience also more memory used.

[NinjaTrader_Jesse]

Hello MojoJojo,

I just wanted to comment on this, while that may work for this use case please be cautious of using that type of code while other scripts are running that you may rely on. It is certainly possible to cause problems by invoking the garbage collection in this type of environment depending on what is going on in the overall platform. I would specifically suggest to avoid this if you are live trading but otherwise you are certainly free to experiment with this and if it helps your use case you can certainly use it for sim purposes.

I look forward to being of further assistance.

[MojoJojo]

Hello Jesse,

thanks for your advice. After collecting some experience with manual garbage collection, it has become apparent that live trading while optimizing is certainly out of the question just like you mentioned. The only benefit is to keep NT from maxing out memory if the parameters are well chosen, however nothing is for free. The much easier and not very expensive solution is to add more RAM.

I forgot to mention in my SampleMACrossOver test that my system has a i7, 4 cores 8 threads, CPU which matters when it comes to memory footprint.