Strategy Analyzer - Freezing during Optimization

[Eros Lima]

Hi,

When Iīm using Strategy Analyzer to Optimize the Strategy, many times stops Remaining Time speed up and freeze the Optimization.

Anyone can help me ?

Regards,
Eros

[Eros Lima]

And Initial Size of Virtual Memory is 256000 MB and Maximum of 512000 MB

[NinjaTrader_ChrisL]

Hi Eros, thanks for your post.

There are a few factors that can cause an optimization to either slow to a craw or just stop. The first one would be the amount of data you are backtesting. Another cause would be if the strategy runs on price change or on each tick with tick replay on, this can exponentially hurt performance. The third cause would be the code itself, something that would need to be optimized by the coder. Another cause would be the type of optimization you are running, performance is negatively correlated with the number of iterations/number of generations you are requesting.

You should use Print statements to print out data from the strategy, this might uncover what is slowing down the system.

If you backtest a small amount of data, does the same thing happen?

I look forward to hearing of your results.

[KarstenKafl]

Hi there

I have been workign with strategy analyzer for some time now.
Im testing with volumetric bars - form 200 tick data - and i have been experiencing some weird differences in speed.
I run same strategy - with same values - only thing i change is time.
I ran test from 1. march to 1. octobver - and it compleeted in 3 - 5 sec. or so.- which was very fast.

Then i ran same test on 1 october 2020 to now in 2021. And then it takes arround 40 seconds to 1.5 miinutes - to "load".

Then i go back and try to load from 1. march to 1. octobver and now it is same speed. 40 sec - 1.5 minutes before it shows results.

Nothign is changed in script - but i think i get this problem some times.

I restartet machine to test - and from now on it is running 40 sec - before i get anywhere on that script.

Im using pc with 64 corers .

Does the time has any influence on this - i mean in or out of the "normal" trading hours? because i dont see i need to load anything from internet - because as far as i can see all data it all is on my harddrive.

Any thoughgts?

[NinjaTrader_Jim]

Hello KrastenKafl,

Thanks for your reply.

There are several factors that need to be considered with optimizations. By nature, they are brute force operations and they will use all of the CPU and memory resources they can. When memory is fully consumed, there will be slight stalls when garbage collection decommits memory and new resources are committed to memory to continue the optimization.

Also to consider, volumetric bars use underlying tick data so long tests will involve downloading a lot of data to build the data series.

If the data series is already cached, loading the data will be faster in the optimization.

I have included some detail below explaining memory utilization in optimizations and some tips to make efficient optimizations.

Explaining Optimization Memory Utilization

I have created a video demonstrating how the SampleMACrossover strategy, a simple strategy that utilizes IsInStantiatedOnEachOptimizationIteration=false for efficiency, can still quickly utilize memory resources.

Demo — https://drive.google.com/file/d/15pz...w?usp=drivesdk

We should consider the following for memory consumption:

Data * Strategy resources * Number of optimization iterations * Number of trades * Keep best # of results.

As we can see there are a number of factors that are involved and memory utilization can climb very quickly depending on a few of these factors. Once memory gets maxed out, we can experience short freezes where memory is decommitted, stored to disk, and then new resources are committed before the backtest is resumed.

We can easily control the number of iterations involved and we can also consider writing our strategies to use IsInstantiatedOnEachOptimizationIteration = false; (Which requires that we reset class level variables in State.DataLoaded.)

IsInstantiatedOnEachOptimizationIteration — https://ninjatrader.com/support/help...niteration.htm

Optimization Tips — https://ninjatrader.com/support/help...ionPerformance

Walk Forward Optimization — https://ninjatrader.com/support/help...ss_metrics.htm

Genetic Optimization — https://ninjatrader.com/support/help..._algorithm.htm

I look forward to assisting.

[KarstenKafl]

Thanks Jim

But i dont see any of this can answer why running the same optimization - on the same script - on the same dates - can suddenly change from taking 5 seconds to taking 60 seconds?
As i wrote - i used it a lot of times - where it ran in 5 seconds - then i chagned the dates, and then it took longer - because it had to load some recent data - but then i changed it back to the "original" dates - and then it took 50 sec. - and even after restarting - and running the same original dates it now takes 50 sec. instead of 5 sec.

[KarstenKafl]

Hi Jim

Again i looked a lot more info this issue - and i understand all you write in your message - but it does not seems to do anything good for me.

So - i noticed few thigns:
1. im runing on 32/64 cores (2 x 8 physical cores)
2. Im running it as multi objective optimization - which for some reason appears to be faster than just running one single backtest... i think becuause of the same issues.
3. When it runs the fast way - (2-3 seconds) - it starts to use full 65-99% cpu - from beginning. The green loading line appears almost instantly while loading.
4. When it runs the slow way - (60 sec +) it starts with white bar - not moving - somehow waiting for something - writing text cacluating - for 50 sec. or so - and then the actual green loading bar appears and now we hit full 65-99% cpu - and the speed is the same as above.
5. I have 64 gb memory - and it uses about 6-7 gb - for running these tests - and it does not change for the two tests. - . .
6. I can make it run slowly - when i just select other dates and return to the same dates - but i have not found a way to make it go back to runing fast. This happens sometime randomly - and sometimes it have been in that "fast" mode - after being left untouched overnight.

When i put in on a chart - with 100 days loaded - on the used 2000 tick volumetric - it never goes above 6% cpu power - but runs for 20 sec. until it is loaded.

Can you say anythign about what the first step of running a strategy does? - i mean - what can it be having problems with - running "Calculations..." in 6% cpu - before it even starts on the actual green bar loading - where it shows "time elapsed" etc.


Thanks.
Karsten

[NinjaTrader_Jim]

Hello Karsten,

The detail is helpful, but I think the best way for me to answer any further questions would be for me to perform the same tests you are with some matching test scripts.

So we can get there, could you set up some tests with the SampleMACrossover strategy, and take note of the exact steps for set up, and questions along each step?

For example:

I have tested the SampleMACrossover strategy with these settings [screenshot attached], and it took X time to complete.

I have then tested SampleMACrossover with these settings and it took X time. Why would this be different?

I have then tested X strategy with Y settings.

If a second strategy is required, please attach what you are using for testing. We would ask the the test strategy be simple for tests. I.E. it would be preferred if we test with SampleMACrossover and a modified version that uses EMA's if separate strategies are required to hit these symptoms. Using a large strategy will add complexity and will make testing and answering questions more difficult. I may not be able to give full insight on a complex strategy that I did not write.

With this detail I will have steps to reproduce the same behavior in an organized fashion, and I would be able to answer specific questions since I can pull up the same.

I look forward to assisting.

[KarstenKafl]

Hi Jim

thanks for your reply.

I have attached a completely ripped version of the script. This script does nothing but loads data - and then does nothing with it.
I also removed all the things that would actually make it "work" - like the BarsArray[1]....etc.

As you can see in the script is is loading volumetric data for 2k tick.

Because when loading that for some reason it takes 1,5 minutes sometimes - and 2 seconds other times - even if i changed nothing.

So when i run this script - it "loads" for 1.5 minutes - with the settings i have entered in this screenshot - looking exactly like the screenshot.
Where it say running calculation - before it starts the usual green "loading" bar.

During the 1,5 minutes it is running at 15 % cpu - and 1-2 gb memory in my original script.

But the weird thing is that sometimes it skips those 1.5 minutes - and starts the green bar - at the second i press the button. This then loads and calculates everything correctly - and shows me the results. But sinse this is much faster - i wish to be able to redo that all the time - but i can not?

When i get it to run fast this happens sometimes after some hours have passed after i last ran it. In fast mode it does the original script calculations in 99% cpu and it is done in very few seconds in total..

These dates and the ES - i believe is speciel for this issue to show.

When i get it to run fast - with the exact same dates and values - if i then go and load other dates - it goes into slow mode again - for the other dates - but also if i then go back to these dates as below image shows. 1 march 20 - to 1. november 20.

I have tested it now on two different pc's.
[ATTACH]n1139630[/ATTACH]

[KarstenKafl]

im not sure - but when i see the post it does not show the attached script - i will just attach ti here as code.

#region Using declarations
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Xml.Serialization;
using NinjaTrader.Cbi;
using NinjaTrader.Gui;
using NinjaTrader.Gui.Chart;
using NinjaTrader.Gui.SuperDom;
using NinjaTrader.Gui.Tools;
using NinjaTrader.Data;
using NinjaTrader.NinjaScript;
using NinjaTrader.Core.FloatingPoint;
using NinjaTrader.NinjaScript.Indicators;
using NinjaTrader.NinjaScript.DrawingTools;


#endregion

//This namespace holds Strategies in this folder and is required. Do not change it.
namespace NinjaTrader.NinjaScript.Strategies
{
public class ScalpOnEMA_RANGE_course_restart : Strategy
{



protected override void OnStateChange()
{
if (State == State.SetDefaults)
{
Description = @"Enter the description for your new custom Strategy here.";
Name = "ScalpOnEMA_RANGE_course_restart";
Calculate = Calculate.OnBarClose;
EntriesPerDirection = 1;
EntryHandling = EntryHandling.AllEntries;
IsExitOnSessionCloseStrategy = true;
ExitOnSessionCloseSeconds = 30;
IsFillLimitOnTouch = false;
MaximumBarsLookBack = MaximumBarsLookBack.TwoHundredFiftySix;
OrderFillResolution = OrderFillResolution.Standard;
Slippage = 0;
StartBehavior = StartBehavior.WaitUntilFlat;
TimeInForce = TimeInForce.Gtc;
TraceOrders = false;
RealtimeErrorHandling = RealtimeErrorHandling.StopCancelClose;
StopTargetHandling = StopTargetHandling.PerEntryExecution;
BarsRequiredToTrade = 20;
// Disable this property for performance gains in Strategy Analyzer optimizations
// See the Help Guide for additional information
IsInstantiatedOnEachOptimizationIteration = true;



}

else if (State == State.Configure)
{

AddVolumetric("ES 03-21", BarsPeriodType.Tick, 2000, VolumetricDeltaType.BidAsk, 1);

}

else if (State == State.DataLoaded)
{

}
}

protected override void OnBarUpdate()
{
if (CurrentBars[0] <= BarsRequiredToTrade || CurrentBars[1] <= BarsRequiredToTrade)

return;


if (BarsInProgress > 1)
{
return;
}





if (BarsInProgress == 1)
{

}

}

#region Properties
[NinjaScriptProperty]
[Range(0, int.MaxValue)]
[Display(Name = "Value", Order = 1, GroupName = "Parameters")]
public int Value
{ get; set; }


#endregion

}
}

[NinjaTrader_Jim]

Hello Karsten,

I just wanted to let you know I have seen your message and the script compiles. I will be doing some testing and will prepare a similar demonstration explaining my findings.

Thanks for your patience.

[NinjaTrader_Jim]

Hello Karsten,

I had thought a video would help to illustrate, but I am not certain what steps you are taking to observe differences in backtest speed.

We should keep in mind that tick data is very granular and there would be significant time added to a backtest or optimization to build that data series. If the data does not exist on the platform it will be requested and downloaded. If we change parameters, the historical data will have to reprocessed to build bars, and then those bars would be backtested/optimized.

This means that there will always be a loading time associated with the optimization or backtest to prepare the data series, even before the backtest starts. This would be where you are seeing longer loading times.

This loading time may be decreased if the data is already loaded on the platform, and the data series has been recently prepared.

If you have any additional questions, please provide steps to follow to reproduce the unexpected fast and slow loading times, and I will be able to comment with more specifics.

We look forward to assisting.

[KarstenKafl]

Hi Jim

Thanks for you reply.

As I tried to explain in my two previous post and image - the big thing here is that it is "random" when it loads for 1.5 minutes - and when it loads for 1 sec. - even with same script, same values and same timeframe etc.

So when i run the script as i wrote above - it loads all data taking 1.5 minutes - i assume that is for getting all the tick data into memory? - and after that it starts the actual calculations of iterations.
It does the same - for 1.5 minutes each time i run the same script - without changin any values - where i woudl assume that all data had already been loaded and was not "Replaced"?

Then suddenly - sometimes if i have been away from pc for some hours - without changing anything - or sometimes if i change it to going through 100 different iterations input values instead of only 10 - then it starts after 1 sec. to do the calculations iterations - using 99% cpu, from beginning.

Then i can change the actual script, and i can change all input values, and i can run it on 1000 of iterations - and it does this witout any "loading" time - only using the time for actual "iterations" - but then if i choose another time frame (so going from running it on the specifc dates i mentioned above - to running it to something from 1. nov. 2020) then it slows down again (this time as expected) - to 1.5 minutes. BUT when i go back to the previous dates i mentioend above - it again takes 1.5 minutes to just get startet - also the second, and third time etc.

And just to be clear - all the database tick data is already loaded on to both my "test" computers that i tried this on.

You write:
"This loading time may be decreased if the data is already loaded on the platform, and the data series has been recently prepared."
How can i see if this is already loaded? - if i run the script once - and then run it again wihtout changin anything - is it loaded then? That is what i do - but with the random results.

You also write:
"please provide steps to follow to reproduce the unexpected fast and slow loading times" I hope i had given all the info - with my image i sent - to at least redo the setup.
If you choose the dates, and run the script - on your machine -with same setup as in my image - how long does it take for you - before it starts to calculate iterations?
If you run it the second time - or third time to see if loading of tick data is already done - on these specific dates - with same setting as in the image - does it then ever get close to 1 or 2 sec. before it start iteration calulation? or is it running for 30 sec - 1.5 minutes - before it starts the actual iteration calculations?

If it does not get close to 1 sec. how can my Pc's do that (randomly)?

Thanks

[NinjaTrader_Jim]

Hello Karsten,

I can't comment on any randomness and would need to follow a procedure to see something specific so I could give a specific reply.

I can say that my tests when connected are generally the same with your strategy or the SampleMACrossover strategy, about 30 seconds testing from the first of the year.

However, I can also say that when I am disconnected that times are much faster. I'm inquiring with our Quality Assurance team since I would expect similar speeds connected or disconnected when the data is already downloaded.

I'll keep you posted after they are able to review my ticket with them. If there is a specific procedure you would like me to follow to look into longer times that is not related to this, I will be happy to test and share my insight there.

[KarstenKafl]

Thank you Jim - it sounds like you experienced the same - and i apreciate it is passed on to the Quality ASsurance team.

Look forward to hearing if there are any ways to avoid the weird extra waiting. Thank.