NT8 Optimizer - Default Algo - Low CPU Utilization

[bunchjesse]

Hi there,

I booted up a 20 core, 2.5GHz VM on the Google Cloud for running my optimizer over a good sized data set. I'm not seeing NT8 use up the available CPUs for the optimization. Here is a sample from the process monitor during an optimization that had been running for about 20 minutes.



Also, the ETA doesn't seem to be accurate:



The ETA counts down (quite quickly) from 24:59:59 to 00:00:00 and then starts over again.

[NinjaTrader_PatrickH]

Hello bunchjesse,

Thank you for your report.

What are the specifics of the test in the Strategy Analyzer?

• Period Type
• Value
• From Date
• To Date
• etc.

Would you have a script that can be used to produce this behavior?

We have seen a report on this but have been unable to reproduce on our side. Any details you have would be most helpful.

[bunchjesse]

It appears that this is caused by having > 24 hours worth of work to do in the optimizer. The logic formatting the time for this label does not take into account periods greater than 24 hours.

Attached is a sample test. Use it on one minute bars of the ES and use "1;40;1" for each of the parameters.

Also, running this test on a 20 CPU machine shows ~20% overall CPU usage and only 3-4 of the cores are in use. It seems that I would expect NT to use all of the available cores when doing this work and I would expect overall CPU utilization to be quite high. Is this not the case? Is NT unable to parallelize this work as much as I wish it could?

Also, here is a screen recording of the issue: http://cl.ly/0F3p3k462B0q

[NinjaTrader_PatrickH]

Hello bunchjesse,

Thank you for your response.

I am seeing all 8 of my cores used for your strategy and parameters in optimization.
In your screenshot it shows all CPU cores at 100% but the graphs do not appear to show the line at 100. Did the graphs eventually show 100 as the percentage showed 100?

[bunchjesse]

I think that dialog is poorly designed. That 100% label is more of a legend showing you what the graph means when it is filled to the top.

See the attached image of the expanded view.

On another note, I've been running an optimization all night and the average CPU utilization has been 30-40% the entire time.

[NinjaTrader_PatrickH]

Hello bunchjese,

Thank you for your response.

NinjaTrader 8 will use one thread per core at the very least, and should utilize the cores as allowed by the operating system. Are you aware of any settings on your system that are specific to threading or core usage?

[NinjaTrader_Matthew]

Since this is a cloud instance, your results vary from the results you would see on a laptop/desktop which Patrick was testing with your strategy.

In my experience with cloud VMs, if you pay for 20 cores, your system will show 20 cores, but that does not necessarily mean you currently have 20 cores dedicated to your instance. I do not know exactly how the Google Cloud instances work, but in general, if you are using scale based hardware profile for your virtual server instance, there is an autoscaler which uses some target average CPU utilization to detect when to add more resources to your instance. As your server environment detected your average CPU increases and you need more CPU to process more tasks, you then have those additional resources to handle some tasks.

However, the way the NT8 SA Optimizer works:
We detect how many virtual cores are available when the optimization is started. If your system only had a fraction of the instances group available to you when the optimization started, there would only be a fractional number of threads spawned off. As your server environment adds more resources, the NT8 optimizer would not re-scale to match what is configured after those resources were available.

There may be some way to set up your server instance group to increase the average expected workload so all resources are immediately available as soon as you start. However, you would need to work with the vendor to understand how that may be configured so you can make the most out of the multi-threaded optimization tasks done by NinjaTrader.

I should also mention that many of these vendors employ quota policies, and if you're just demoing a server, you may not have the full quota available during the time you're testing. If you pay for a server, depending on how your resources are being used throughout a time period, you could go over your quota, at which point the vendor would restrict access to additional hardware, and consequently limit the number of threads that the optimizer can utilize.

[NinjaTrader_Matthew]

It appears that this is caused by having > 24 hours worth of work to do in the optimizer. The logic formatting the time for this label does not take into account periods greater than 24 hours.

Attached is a sample test. Use it on one minute bars of the ES and use "1;40;1" for each of the parameters.

Also, here is a screen recording of the issue: http://cl.ly/0F3p3k462B0q

Thank you for this - We've reported to development and is tracked using NTEIGHT-9684

[mrengleman]

I have had similar problems with NT8 not fully utilizing all instances on different cloud servers and I do not think this is due to quota policies, ect because I've tested other multi-threaded trading programs on the same cloud servers(AWS and smaller VPS companies) and seen 100% utilization.

I know there will be some variation between in utilization between a personal computer and cloud servers but 100% utilization on a personal computer vs 30% variation on a cloud server is not a normal variation.

Please don't automatically assume the lower utilization is due to the cloud environment, and not NT8.

As your server environment adds more resources, the NT8 optimizer would not re-scale to match what is configured after those resources were available.

Well, this sounds like then that the optimizer is poorly designed and too inflexible. What good is a multicore optimizer if it's only uses multiple cores if certain conditions are met? Have you published what those conditions are?

It's not unreasonable to expect software running computationally intensive tasks to be able to take advantage of the power of 20+ core cloud servers.

[NinjaTrader_Matthew]

I have had similar problems with NT8 not fully utilizing all instances on different cloud servers and I do not think this is due to quota policies, etc because I've tested other multi-threaded trading programs on the same cloud servers(AWS and smaller VPS companies) and seen 100% utilization.

Please don't automatically assume the lower utilization is due to the cloud environment, and not NT8.

The cloud servers in this scenario work exactly as they were designed. I did not intend to somehow point to the cloud server be the issue.

To reemphasize from my last post, the issue is with the way optimization threads are collected on startup in NT8. Until we might be able to improve this scenario for cloud instances, you may be able to work around this designed behavior by

a) allocating more resources in your cloud instance before NT8 is started
b) restart NinjaTrader 8 once your resources have been allocated so the SA can take full advantage of the cloud servers.

Have you published what those conditions are?

There is an indefinite number of computer performance related tips/etc product wide which have not been completely finalized, but we will be adding more to the help guide and document them we conclude performance testing throughout beta.

Specific to the strategy analyzer optimizer, we have some notes collected on the link here below and we will be adding more information as we see fit. I will make a case for publishing those conditions here. Thanks for your feedback.

http://ninjatrader.com/support/helpGuides/nt8/en-us/index.html?optimize_a_strategy.htm#UnderstandingFactorsThatAffectOptimizationPerformance