NT8 Low CPU Utilization

[Conan108]

While running Strategy Analyzer Optimization of the Sample MA crossover strategy on a 4-core AWS cloud instance (t2.xlarge), NT8 is only utilizing 1 of the 4 available CPU cores. As you can see from the attached screenshot, Windows Task Manager is aware of 4 CPU cores. (You can see there are 4 cores available because there are 4 CPU Usage graphs in the Windows Task Manage. Additional details about the Hardware Resources and Operating System of the AWS instance and are also displayed in the screenshot.) So, even though Windows Task Manager was aware of 4 available CPU cores, NT8 was apparently only aware of one of them.

This post, suggests that some multi-core cloud instances only allocate additional CPU cores as they are needed (dynamically), which prevents NT8 from becoming aware of additional CPU cores because NT8 only checks for # of available cores once, when it is started. But in my test case, I opened Windows Task Manager before starting NT8 and visually confirmed that Windows Task Manager was seeing 4 available CPU cores before NT8 was started.

Addressing the same issue later in the same thread, in this post mrengleman makes some very good points and poses a very important question. The first important point he makes is this one:

Please don't automatically assume the lower utilization is due to the cloud environment, and not NT8.

Then he references a statement Originally Posted by NinjaTrader_Matthew:

As your server environment adds more resources, the NT8 optimizer would not re-scale to match what is configured after those resources were available.

mrengleman's excellent and very insightful response to Matthew's statement is:

Well, this sounds like then that the optimizer is poorly designed and too inflexible. What good is a multicore optimizer if it only uses multiple cores if certain conditions are met? Have you published what those conditions are?

That question about whether the specific conditions necessary for utilization of multiple cores have been published was addressed later in the same thread by NinjaTrader_Matthew in this post, where he says:

There is an indefinite number of computer performance related tips/etc product wide which have not been completely finalized, but we will be adding more to the help guide and document them we conclude performance testing throughout beta.

Specific to the strategy analyzer optimizer, we have some notes collected on the link here below and we will be adding more information as we see fit. I will make a case for publishing those conditions here. Thanks for your feedback.

http://ninjatrader.com/support/helpG...ionPerformance

However, in reading through the help guide page referenced by NinjaTrader_Matthew, I find no information published there about the specific conditions necessary for full utilization of multiple cores.

Thus, I find myself in a pickle - really wanting to use multiple CPU cores while running Strategy Analyzer Optimization, but unable to do so. If I go to NT tech support for help, they will likely say, "AWS is not setting the proper flags/variables in the Operating System to let NT8 know how many CPU cores are really available." But before I can go to AWS tech support to request that they set the proper flags/variables in the OS, I need to know exacltly which flags/variables in the OS need to be set. Otherwise, AWS will likely tell me it's NinjaTrader's problem, not theirs. So, I find myself reiterating mrengleman's request to NinjaTrader to please publish the specific conditions which need to be met for NT8 to be able to utilize all available CPU cores. Or, better yet, make NT8 smart enough that it doesn't require any special conditions to be set in order to utilize all available CPU cores. In my test example, if NT8 could detect the number of available CPU cores in the same way Windows Task Manager detects the number of available cores, then NT8 would be able to utilize all available CPU cores. Is there any reason why NT8 can't just duplicate whatever Windows Task Manager is doing to detect the number of available CPU cores?

Please feel free to reproduce the test example illustrated in the attached screenshot. All the parameters used in the optimization of the Sample MA crossover strategy are visible in the screenshot. I'm guessing NinjaTrader has an AWS account, but if not, I'd be happy to let them use my account to create a "t2.xlarge" instance of "Windows Server 2008 R2 Datacenter" so they can reproduce the problem I'm seeing.

Note, in my test case above, I did not change the NT process CPU Affinity or the NT process Priority; they were both left at their default settings.

Thanks in advance for whatever information anyone has on this issue.

Regards,

Conan

Note, I had to post this under a new thread, different from the one referenced above, because that thread is in the "NT8 Beta" section, which was closed to further posts when NT8 went out of Beta.

[NinjaTrader_PatrickH]

Hello Conan108,

Thank you for your post.

I will follow up with our Product Management team on this matter and follow up here with you on the details we find.

I look forward to assisting you further.

[NinjaTrader_PatrickH]

Hello Conan108,

Thank you for your patience.

This appears to be a different case then the one you are referencing, although being quite similar. In your case the cores are there when starting.

Can you send me your log and trace files so I may look into this matter further?
You can do this by going to the Control Center-> Help-> Mail to Platform Support. Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default. Please list 'ATTN: Patrick H' on the subject line and reference this thread in the body of the email.

I look forward to your response.

[Conan108]

For the benefit of those interested in this thread, here's the resolution of the issue, as described in my last email to Patrick:

Hello Patrick,

Thank you so much for the additional details. The information you provided about how NT8 detects the number of available CPU cores, and where it displays the number it detected, has allowed me to do some additional debug work, and I’m happy to report that I found the bug.

You may remember that, in a previous email, I mentioned that NT8 had suddenly and unexpectedly started using both CPU cores on a 2-core AWS instance, whereas before it was only using 1 of the 2 cores. I eventually figured out that I had originally installed NT8 on a 1-core instance of AWS (t2.small). Then, later on I used a feature of AWS which allowed me to upgrade the instance from 1 core to 2 cores (“Instance Settings—> Change Instance Type”). But unfortunately, NT8 currently only checks the number of available cores at installation time, not at launch time. So, NT8 was only aware of 1 core until one day I re-installed NT8 on the 2-core version of the instance. Then suddenly, NT8 was aware of both cores.

In the old days, when the number of available cores didn’t typically change after a SW package was installed, detecting the number of CPU cores on installation time worked perfectly. But now, with the advent of cloud computing and the ability to resize a cloud instance, the number of available cores can change after a SW package has been installed.

To confirm my theory, I ran some more tests. First, I duplicated what you showed in your video. I installed NT8 on a t2.xlarge, 4 core instance, and NT8 successfully detected and utilized 4 out of 4 available CPU cores. Then I stopped that instance and resized it to just 1 core (t2.small). When I launched NT8 on the 1 core, t2.small instance, the NUMBER_OF_PROCESSORS environment variable was set to 1, as expected, but the NT8 trace file showed that NT8 had identified “PROCESSORS = 4.”

I also tested resizing an instance to a larger size. NT8 was installed on a 2 core instance, which was then upgraded to a 4 core instance. In this case, on the 4 core instance, NUMBER_OF_PROCESSORS was set to 4, as expected, but the NT8 trace file had only identified “PROCESSORS = 2.”

So, there is no doubt that NT8 is detecting the number of available processors at installation time, not at NT8 launch time. As I said, this worked fine when the number of available processors never changed after an NT8 installation, but it doesn’t work in today’s cloud computing world.

Please pass this discovery on to the development team so they can move the code to detect the number of available processors from the installation code area to the launch code area. In the mean time, I’m going to post this reply to the forum thread I initiated on this subject so that others can work around this NT8 limitation until it’s fixed.

The developers may be tempted to think it will be extremely unlikely that someone will want to upgrade or downgrade some NT8-installed instance to a different number of CPU cores. However, I find myself wanting to do just that all the time. I can save $$ by using a low powered instance while backtesting specific time periods of interest. Then, when I’m ready to run the Strategy Optimizer, I want to upgrade the number of CPU cores for a faster optimization run. Unfortunately, the only way I can think of to accomplish this with the current NT8 limitation is to create two separate AWS instances, one with many cores and one with few cores, then install NT8 on both of them. Unfortunately, this means I’ll need to copy the latest code and the latest parameters from one instance to the other each time I switch between them. Hopefully it won’t take long before this limitation can be fixed in NT8 so I and others won’t have to work around it like that.

Regards,

Conan

[sledge]

To confirm my theory, I ran some more tests. First, I duplicated what you showed in your video. I installed NT8 on a t2.xlarge, 4 core instance, and NT8 successfully detected and utilized 4 out of 4 available CPU cores. Then I stopped that instance and resized it to just 1 core (t2.small). When I launched NT8 on the 1 core, t2.small instance, the NUMBER_OF_PROCESSORS environment variable was set to 1, as expected, but the NT8 trace file showed that NT8 had identified “PROCESSORS = 4.”

I also tested resizing an instance to a larger size. NT8 was installed on a 2 core instance, which was then upgraded to a 4 core instance. In this case, on the 4 core instance, NUMBER_OF_PROCESSORS was set to 4, as expected, but the NT8 trace file had only identified “PROCESSORS = 2.”

So, there is no doubt that NT8 is detecting the number of available processors at installation time, not at NT8 launch time. As I said, this worked fine when the number of available processors never changed after an NT8 installation, but it doesn’t work in today’s cloud computing world.

Please pass this discovery on to the development team so they can move the code to detect the number of available processors from the installation code area to the launch code area. In the mean time, I’m going to post this reply to the forum thread I initiated on this subject so that others can work around this NT8 limitation until it’s fixed.

That's pretty wild.

It would be pretty easy on my Intel I7-5820k to test, by turning off hyper threading in the BIOS.

Currently "12 cores" are detected, so if I turned it off - it should wrongly detect 12, instead of 6.

I might try that tomorrow.

A clean install should detect 6, and then turning back on hyper threading should wrongly display 6.

I only use Playback-MR, so it may not matter in my situation.


--Confirmed--
I disabled hyperthreading in BIOS, and NT8 "detects" "12 cores", instead of 6.
CPU-Z reports 6 cores, 6 threads.


In your Ninjatrader 8 folder - edit config.xml in an XML editor or notepad.
Find "NumberofWorkerThreads", and adjust accordingly. I just bumped mine to 24, and the trace file is reporting this #.

[NinjaTrader_MichaelM]

Hello everyone,

As an update to this issue, it was determined that certain parameters were being serialized/deserialized in user's Config.xml files improperly, including NumberOfWorkerThreads.

This has been resolved for the upcoming R10 release.

For any users who run into this issue in the meantime, sledge is correct however I would suggest instead removing the entire tag line:

<NumberOfWorkerThreads></NumberOfWorkerThreads>

from your config file, saving it, and then launching NT8 so it can properly re-detect the number of workers.

Thanks!

[Conan108]

Thanks, Michael, for the update, and for getting this issue fixed in the next release of NT8.

On a related point, I've recently done some testing using cloud instances with various numbers of CPU cores. Both Amazon and Google now offer compute instances with up to 64 CPU cores. Unfortunately, when I run Strategy Analyzer Optimization on a compute instances with large numbers of cores, the performance does NOT scale with the number of cores. For virtual machines with less than about 15 cores, each core seems to be highly utilized, and the overall CPU utilization of the machine hovers around 80-90%. However, when performing the same optimization run on say, a 64 core machine, each core is barely utilized and the overall CPU utilization of the machine is less than 25%, often MUCH less.

Apparently, there is some resource contention issue which prevents the performance from scaling with the number of available cores. Less than half the available physical memory was being utilized in my test of the 64 core virtual machine, so the problem was not a lack of physical memory availability. Maybe contention between the threads over access a specific static memory location, or a specific file, is creating a bottleneck?

I'd be happy to share my strategy with NT support if they need it to confirm this behavior, but I've seen similar behavior on different strategies. I think I even saw it on the Sample MA Crossover strategy, so that would be a good place to start trying to replicate my findings.

I'd be curious to know if others have had similar experiences.

Regards,

Conan

[NinjaTrader_PatrickH]

Hello Conan108,

Thank you for your post.

We should not need your strategy as this would be related to the application itself. Can you please send us your log and trace files?

You can do this by going to the Control Center-> Help-> Mail to Platform Support. Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

Please list 'ATTN: Patrick H' in the subject line and reference this thread in the body of the email.

I look forward to your response.

[NinjaTrader_PatrickH]

Hello Conan108,

We have not received your log and trace files. Please follow the steps below to manually attach your log and trace files to an email to platformsupport[at]ininjatrader[dot]com so I may investigate this matter further.

• Open your NinjaTrader folder under My Documents.
• Right click on the 'log' and 'trace' folders and select Send To> Compressed (zipped) Folder.
• Send the 2 compressed folders as attachments to your email.
• Once complete, you can delete these compressed folders.

Please list 'ATTN: Patrick H' in the subject line and reference this thread in the body of the email.

I look forward to assisting you further.

[PN720]

Hey fellas,

Just following up on Conan108's note about running Strat Analyzer on multiple core machines..

I also tested on Amazon AWS with 8 vCPUs - 16GB Memory, 16 vCPUs - 32GB Memory, and 72 vCPUs - 144GBMemory and noticed that the 16 and 72 instances didn't show scaling appropriate to the number of CPUs.

The 8 vCPU one went up to 80-90% CPU usage but the 16 vCPU topped out around 58% and the 72 vCPU didn't even get above 20%.

I have the log files form the the 8 and 16 ones, if you are interested.

[NinjaTrader_PatrickH]

Hello PN720,

Thank you for your post.

Please do send us over the Log and Trace files for this instance.

You can do this by going to the Control Center-> Help-> Mail to Platform Support. Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

Please list 'ATTN: Patrick H' in the subject line and reference this thread in the body of the email.

I look forward to your response.