reduce memory after backtest

[Amedeus]

Hello everyone,

I have a question about memory and performance.

The memory column from the task manager is around 60% when the platform is on, at the opening of the app when all charts are loaded and working.
After a backtest it is around 85-90%.
After a few more backtests it is around 95%, and does not fail below 90%.

It was around 97% with regular freezes of the app, then I limited the size of all the lists from the code, reduced also the demand from the loaded indicators.
And it works better now, but still : I would like the memory to come back to the level before the 1st backtest.

Even if I tell the command : ipconfig /FlushDNS, the memory doesnt go much lower, not below 85% anyway.

Has anyone any idea about making the memory level go where it was before the backtets ?
what is the smart move when confronted to this ?
Should GC.Collect(); be involved and how/where ? I did put it in State.Terminated without success, concluded the memory used is not GC...

[NinjaTrader_Jim]

Hello Amedeus,

Thanks for your post.

Memory is freed when Garbage Collection takes place. It would not be necessary to invoke garbage collection on your own, but you could consider marking objects as null so they can be more readily picked up by Garbage Collection.

Please see our Performance section under Best Practices for more information and other performance tips.

https://ninjatrader.com/support/helpGuides/nt8/index.html?ninjascript_best_practices.htm#Performance

Let us know if there is anything else we can do to help.

[Amedeus]

Hey Jim, thank you very much.

In State.Terminated I did mark as null :
- all the lists,
- all the indicators instantiated in State.DataLoaded.
There are no drawing objects, or brushes in this script, there are quite a few variables which we cannot mark as null...

So, marking this above as null helped, but what if the memory level still does not go below 80% ?
And do you know of a way to output the objects in case I missed any ?

[NinjaTrader_Jim]

Hello Amedeus,

If you are making moves to null objects so they can be marked for garbage collection, you would be making the moves that you can within the script to free resources.

Data may be held onto in memory to aid in faster data loads, which would start to be cleared when available memory becomes sparse.

On a similar topic, there are some things to keep in mind with memory utilization and optimizations. Tips on running memory efficient optimizations can be found below.

https://ninjatrader.com/support/helpGuides/nt8/index.html?optimize_a_strategy.htm#UnderstandingFactorsThatAffectOptimizationPerformance

[Amedeus]

Hey Jim, thank you very much.

There have been some positive changes after testing some of the tips.
But we still are oscillating between 80% and 90%.
With a drop towards 80% at the start of the backtest iteration, ending around 90%, staying at 90% until the next backtest (then drop to 80 when new backtest, and so on)
If this pattern tells something specific about what to do in order have the memory level fall, please let us know.


Another question came up :
I see no difference in the trade analysis (pnl) and AllTrades.Count when setting IncludeTradeHistoryInBacktest to true or false.
Is this expected ? does IncludeTradeHistoryInBacktest=false includes some of the trades history after all ?

[NinjaTrader_Jim]

Hello Amedeus,

I don't have any other tips to advise for clearing your own strategies resources. If you are minding best practices, it is likely data and trades that would be in memory that are not immediately cleared.

Regarding IncludeTradeHistoryInBacktest:

When I test the attached script which sets IncludeTradeHistoryInBacktest=false in State.Configure, I see the results I would expect. SystemPerformance.AllTrades.Count is 0, and I do not see any trade lines connecting execution markers on my chart.

[Amedeus]

hey Jim, thanks a lot again.

Of course I had IncludeTradeHistoryInBacktest=false in state.SetDefault instead of Configure.

"it is likely data"

Does this include added dataseries in a mtf strategy ?
those could definitely be a big suspect.

What would be the declaration syntax to make them objects ? that we would mark as null later on.
I tried something like this below, but we can not convert a type "void" to "ninjaScript.series<Ninjatrader.Data.Bars>"

private Series<Bars> dts1;

protected override void OnStateChange()
{
...
else if (State == State.Configure)
{
dts1 = AddDataSeries(Data.BarsPeriodType.Tick, intOfTicks);
}
...
else if (State == State.Terminated)
{
dts1 = null;
};
}

would such stunt be pointless because added series would be collected by GC ? or because of some other reason ?

[NinjaTrader_ChelseaB]

Hello Amedeus,

AddDataSeries() does not return an object. This only adds a data series. (void means no object is returned)
https://ninjatrader.com/support/help...dataseries.htm

The code below is invalid.
dts1 = AddDataSeries(Data.BarsPeriodType.Tick, intOfTicks);

The data is added to the BarsArray collection.
https://ninjatrader.com/support/help.../barsarray.htm

The BarsArray collection is handled by NinjaTrader behind the scenes to drive the script and cannot be set to null.

Yes, added series will take memory for the data, and will use CPU resources when running for that BarsInProgress in OnBarUpdate().

Custom series variables, on the other hand, can be set to null as these objects are not used to drive the script.
https://ninjatrader.com/support/help...t8/seriest.htm

[Amedeus]

Hey Chelsea and Jim, thank you very much.
What's been marked as null made a big difference.
have a great week.