memory leaks in strategy run by Optimizer?

[momchi]

Hi,
please point me to articles or hints what could be causing memory leaks when my strategy is run in Optimizer?

Ninja Trader8 starts with roughly 480 MB in RAM, then at each run of Optimizer it adds another 200-300MB consistenly, until my RAM usage is 98% at which point I have to kill the process and start again... this happens after about 10 runs ... what might be causing this?

IsInstantiatedOnEachOptimizationIteration = true
changing it to false does not help

In State.Configure I do add a secondary data-series and two float series for a custom indicator:

// add minute data series to watch for EOD
AddDataSeries(BarsPeriodType.Minute, EODPeriod);


// input data series that AAAAAAAAindicator will use to plot in indicator's panel
// we supply this data here, as opposed to a standard indicator which uses bar data as input
IndicatorLong = new Series<double>(this);
IndicatorShort = new Series<double>(this);

then I fill up the above with historical data in State.DataLoaded, then update at each OnBarUpdate() with the new bar value:
(example)
IndicatorShort[0] = XXXXX.Value[0];
IndicatorLong[0] = YYYYY.Value[0];

Should I release/free all this data somehow after each run to avoid this piling up of memory usage?

thanks in advance,
Momchil

[MojoJojo]

Hi,

somewhat related, also a problem is that the strategy analyzer requires a whopping 7GB of RAM on my computer without having any test run(latest NT version).
To be sure that it's not caused by any of my code, I disabled all my indicators and strategies by switching them to mock classes that extend NinjaScriptBase but not Indicator or Strategy so NT doesn't find them.

My test procedure was:
1.) open NT -> 160MB with one Market Analyzer running
2.) open Strategy Analyzer -> 10.5 GB
3.) SampleMACrossOver optimization -> 7GB

It was able to free up 3GB, but I'd curious why the analyzer requires a long startup time and RAM. NT managed to fully occupy all my free RAM on previous tests.
I can only theorize that back testing and optimization don't need to preload a lot of data. If it turns that a probably rarely used feature like the AI needs all these resources would it be possible to split it off from the strategy analyzer?

I am not sure it's somehow related to the latest version of NT, Maybe I'll give it a try to revert back and will report later if the situation improves.
However I was able to run some optimizations with the current version earlier so I'd guess it's possibly not caused by new code.

[NinjaTrader_Jim]

Hello Momchil,

Thanks for your post.

When using IsInstantiatedOnEachOptimizationIteration = false, the strategy must reset all class level variables in State.DataLoaded to ensure that left over values are not used since the script will not be instantiated for each optimization iteration. This would be advised for best performance when backtesting.

Garbage collection will take place when NinjaTrader needs memory. We should expect that memory will be freed periodically when the strategy is running the optimization and the platform should not crash.

With regards to factors impacting memory load, heavy usage of lists and similar objects can do this, strategies that create large number of trades can increase this, and using high values for Keep Best # of results will increase the amount of data that is kept in the optimization.

momchi I would suggest ensuring that your script is compliant with IsInstantiatedOnEachOptimizationIteration = false, and I would suggest approaching this from two angles:

1. Test to reproduce the issue from a clean start of the platform with workspaces closed so we know we are starting at a clean point and so we can focus on where memory is getting spent. I would suggest then to reduce parts of your strategy code to identify parts that are consuming memory. Keep track of the settings used as well. Between tests, please make sure to restart the platform so we are testing from a clean slate.

2. Keeping track of the settings used, attempt to reproduce the symptoms using the same settings, but with the SampleMACrossover strategy. This should help to rule out if memory consumption is high for your specific strategy.

MojoJojo I do not see these symptoms when I open NinjaTrader and then open a Strategy Analyzer. Memory usage stays low until I run an optimization when I take the same steps you are. Could you reach out to me at platformsupport [at] ninjatrader [dot] com so we can further discuss this? I would like to keep this forum thread on topic with momchi's specific inquiry. Your symptoms seem a little different. Please include the text "2588178 Attn Jim" so the ticket gets assigned to me.

I look forward to assisting.

[MojoJojo]

Just a quick and last update.
For people who performed a lot of optimizations, after deleting the content of %HOMEPATH%\Documents\NinjaTrader 8\strategyanalyzerlogs, everything is back to normal.

[momchi]

Hi Jim, thank you for the reply -
as I am working through suggested steps, a question: what states a strategy goes through at the start of a new optimizer step when IsInstantiatedOnEachOptimizationIteration = false?
In particular these are of interest to me: Configure, DataLoaded and SetDefaults.

[NinjaTrader_Jim]

Hello momchi,

Strategies will process Configure, DataLoaded, and then Historical when a new optimization iteration takes place. We suggest resetting class level variables in State.DataLoaded for a single state to perform these operations since variables based on loaded data could not be reset in State.Configure.

Let us know if you have any additional questions.

[momchi]

Hi Jim,
sorry but do need to come back to this topic - I did move all the resetting of class variables in DataLoaded, as suggested. Still, with every Optimization Run the memory used by NinjaTrader grows steadily by a certain amount - around 200-250MB per run of 1 week data (1min + 10,000 ticks).

I do have a custom indicator that is created/instantiated in Configure state, and the added to the Chart in DataLoaded state. But even if I don't add it to the chart (still use it's computed data though), the memory heap grows.

To save memory when in Analyzer/Optimizer, I now have to remove all log printing and all drawing objects (that indicate my signals on the chart) - wish there was a way to force the "garbage collection" between the Optimizer runs... Is there?

Unfortunately, I was not able to make my strategy work when IsInstantiatedOnEachOptimizationIteration = false, cannot to avoid data series errors (accessing non-existent index).

best regards,
Momchil

[NinjaTrader_Jim]

Hello Momchil,

The strategy will need to programmed to use IsInstantiatedOnEachOptimizationIteration = false in order to improve backtesting speed and memory utilization. Leaving this set to true will be less efficient with computer resources.

I would suggest running a couple small backtests for a "control" test and then to also test an optimization over those two parameters. This can help so the errors are reproduced and chase-able, and so you can have a frame of reference for what the expected behavior for that backtest will be when it is not performed in an optimization. You could then use prints to check the strategy behavior line by line, and then correct each error as you run into it.

If you are stuck on a specific error or have a question on a specific part, you can prepare a test strategy that includes just that part and share it with us so we can give further advise on how that error can be debugged.

There is not a way to force garbage collection, and as long as we are not creating optimizations that will hold a large number of "Keep # of best results" and we run efficient backtests (avoid iterating every possible combination of parameters) we should be able to avoid hitting memory caps.

Additional tips for improving optimization performance that we should keep in mind can be found here - https://ninjatrader.com/support/help...ionPerformance

We look forward to assisting.

[momchi]

Hi Jim,
thanks for the guidance here and advice! Yes, I realized that if I wanted to use the switch above not to instantiate it at every iteration, I needed to do a better job initializing my data series and arrays. Will try what you suggested and hopefully come back with some log files + more specific questions.
Best regards,
Momchil