Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Limits of Optimizer Multi-Threading

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Limits of Optimizer Multi-Threading

    Are there any limits on the Multi-Threading capabilities of the Ninjatrader Optimizer? If I was to upgrade my CPU to a 32 Core, Like the new Ryzen Threadripper 2990WX will the optimizer be able to make use of all of the cores; or is there a limit on the number of threads that can be utilized?

    Has any benchmarking been done at massively multi-threaded optimization? Does memory become a bottleneck or any thread starvation due to locked resources etc?

    Just wanted to check if anyone has any info on this before I spend $2k on a new CPU.
    Last edited by kevinenergy; 10-09-2018, 01:18 PM.

    #2
    Hello kevinenergy,

    Thanks for your post.

    NinjaTrader uses a thread pool for processing historical data and creates threads in line with the number of logical cores on your CPU. Multiple threads helps greatly with optimizations. I am not aware of any noted profiling done and I am not aware of particular bottlenecks on a large scale.

    I do have similar hardware on my home PC. I'll update this post with some comparisons using a test script and share my results when I get a chance.

    Please let me know if there is anything we can do to assist.
    JimNinjaTrader Customer Service

    Comment


      #3
      That would be really helpful - thanks you Jim! It doesn't need to be a comprehensive test. But what I'd like to know is: if you switch the performance Tab on Task Manager to "Logical Processors" as below; is the Optimizer still pushing each one to 90% with a huge number of threads running. I'm concerned that maybe the north bridge will become a bottleneck with so many threads pulling in data from RAM which would be the likely cause if the CPU utilization is lower.

      Thanks again for offering to test this for me!! Really appreciated!

      Comment


        #4
        Hello kevinenergy,

        I ran a quick optimization this morning using SampleMACrossover with Fast 1;200;1 and Slow 1;200;1 against 1 minute data since the beginning of the year on ES. Really just wanted to max out CPU usage with a simple script so number of iterations could be focused on (since you are inquiring about thread usage.) All my logical cores are being utilized pretty well, and max CPU usage does even out between 70% and 90%. I'm sure a more complex strategy will keep overall utilization at a higher percent. Memory is also being heavily utilized, but I do have other applications running.

        Here's a screenshot:

        Click image for larger version  Name:	Taskmgr_2018-10-10_05-54-10.png Views:	1 Size:	146.5 KB ID:	1035121

        Please let us know if there is anything else we can do to help.
        Last edited by NinjaTrader_Jim; 10-10-2018, 07:27 AM.
        JimNinjaTrader Customer Service

        Comment


          #5
          Jim! Thank you for taking the time to do this. Looks like a good result to me. How fast is your RAM in that rig if you don't mind me asking?

          Comment


            #6
            Why not do this in the Cloud, such as Google or AWS? I do extensive optimizations and gotten very good results. GC offers 96 cores.

            Comment


              #7
              I looked into it. But $4.50/hr quickly adds up. That's $800/week.

              Originally posted by spottysallrite View Post
              Why not do this in the Cloud, such as Google or AWS? I do extensive optimizations and gotten very good results. GC offers 96 cores.

              Comment


                #8
                Hello kevinenergy,

                The RAM is earlier DDR4 RAM. I don't remember the speeds off the top of my head, and I would like to leave this thread in the direction it has taken rather than talk about PC specs.

                If there is something else we can assist with, please feel free to open a new thread.
                JimNinjaTrader Customer Service

                Comment


                  #9
                  Reason I was asking is I wondered if I could squeeze the Optimizer up to MAX core utilization with faster RAM. Earlier DDR4 so probably 2800-3000Mhz. There's 4200Mhz availabe now so hopefully that will squeeze even more out of it. Wasn't just generally discussing PC specs, don't worry haha. I'm not THAT much of a geek.

                  I will update this thread when I get my new PC and have done some benchmarking.

                  Comment


                    #10
                    The optimal number of cores for me seems to be around 16, and that's using 5 years of single-tick data on the ES, and optimizing 27 params with 3 values each.

                    How about spending something less than $2k and overclocking and cooling using an isopropyl alcohol / dry ice bath?

                    Comment

                    Latest Posts

                    Collapse

                    Topics Statistics Last Post
                    Started by onnb1, Today, 02:08 PM
                    0 responses
                    2 views
                    0 likes
                    Last Post onnb1
                    by onnb1
                     
                    Started by omermirza, Today, 01:15 PM
                    2 responses
                    9 views
                    0 likes
                    Last Post omermirza  
                    Started by stevenev1, Today, 01:24 PM
                    1 response
                    8 views
                    0 likes
                    Last Post NinjaTrader_BrendanC  
                    Started by morrisonrob, Today, 12:59 PM
                    1 response
                    7 views
                    0 likes
                    Last Post NinjaTrader_Kate  
                    Started by JTizz, Today, 01:00 PM
                    1 response
                    8 views
                    0 likes
                    Last Post NinjaTrader_BrandonH  
                    Working...
                    X