Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Does NT8 use all Cores/Threads when backtesting?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Does NT8 use all Cores/Threads when backtesting?

    Does NT8 use all the computer resources when optimizing / Strategy Analyzer?

    I have an 8 Core / 16 Thread CPU, X570 Mobo and SSD. When looking at the Task Manager it appears that NT8 is only using 2 Cores / 4 Threads when running an optimization. This was confirmed when I looked at AI Suite 3. It does not appear to be Thermal Throttling as temps are nowhere close to the danger zone. Total CPU Utilization is always under 10%. Leaving such a large percentage of CPU Utilization on the table seems like a colossal waste of time. Is this normal? Or has something gone wrong with my system? I also noticed the same thing when using Market Analyzer for an EOD Stock Screener. I have never noticed this before.

    If someone wants to try to duplicate I was optimizing the stock strategy 'Sample MA crossover' on the SP500 Select All.

    #2
    legitimate question. Although Ninjatrader claims to be fully multithreaded and 64bit capable I'm surprised that the application is installed and running under "C:\Program Files (x86)\NinjaTrader 8". Normally in Windows all 64bit demanding applications are installed under C:\Program Files\ by default, so the correct 64bit installation folder for Ninjatrader8 64bit should be "C:\Program Files\NinjaTrader 8".

    I am not sure if NT8 is capable of using all cores on a system. Certainly you will get a reply from a Ninjatrader support member. But maybe and hopefully some hardware tech guys can provide some more additional information how it's possible to stress test NT8 and see if it really makes use of all available cores of the computer system. So you could make a simple test on your own and proof if NT8 really is multi-threading capable.

    Comment


      #3
      Om my trading workspace it does appear to be using all core / threads. ES with 3 charts plus one more with 4 tabbed charts for a total of 7 charts. It is the Market Analyzer and Strategy Tester that I am questioning. Rarely does the Total CPU Utilization get above 7%with all cores/threads showing activity on the Trading Workspace..

      Comment


        #4
        how do you know, how did you proof that Ninjatrader8 utilize all your cores with full available CPU speed and 100% ?

        In my case it certainly does NOT utilize all the available 8 threads on my system. When I run a stragegy analyzer on any strategy with lots of historic data and computation power needed, it takes a long time until it finishes. This is very unusual because I own a very powerful system. Running 8 cores @4GHz Intel CPU with 32GB RAM.

        I opened Windows10 CPU detailled task manager. Press CTRL+ALT+DEL --> Task Manager --> tab PERFORMANCE and then on the bottom click on "Ressource Manager" which will open a more detailled view where you can select more options.

        Click image for larger version

Name:	win10_ressource_manager.png
Views:	71
Size:	208.7 KB
ID:	1128821

        Click on the CPU tab at the top pane. Now at the right window area you will see all your available threads of your machine and its current utilization. Run something heavy computing task in Ninjatrader and watch at your cores/threads. In my case even the CPU speed remains throttled at energy saving setting @1.6GHz !!! remember, my system is capable of utilizing 4GHz per core. I can run CPU stress testers or even the Intel processor diagnostic tool with its own CPU benchmark and stressing tool and in all cases all my threads are 100% utilized, that's expected behavior.

        But Ninjatrader8 never ever is utilizing my 8 threads, it even does not utilizing 100% of one single thread. That explains why my NT8 takes sooooo long for computing and charting things. That's very dissapointing. For the pro users our there, you can even download and run the free available "Process Explorer" from former Sysinternals (now owned by Microsoft) to get even more detailled information about how Ninjatraders is utilizing your systems ressources. You can click on running NinjaTrader.exe process and show what exactly this process is performing on your system. Again, I see very poor results because it does not utilize the available hardware ressources of my system. I doubt, that Ninjatrader8 is true 64-bit and multi-threading capable after that experiencing.

        In any case I forgot something, please let me know. I search but did not find any certain setting that would be necessary to enable in order to allow Ninjatrader to utilize multi-threading on systems. Usually one would expect that the application in use should automatically use the available threads and cores. If there is any hidden setting in Ninjatrader, please let us know.

        Looking forward to your check and comments.

        Comment


          #5
          Hello TAJTrades,

          NinjaTrader 8 is multi-threaded and uses dispatcher thread pool control. From there, the windows operating system designates which logical processor (CPU core) any thread will be run on and the operating system will handle load balancing.

          Data, orders, backtests, optimizations, are all processed separately in different threads than the consumer script's thread.

          Below is a link to the help guide on factors that affect optimizations.
          https://ninjatrader.com/support/help...ionPerformance

          An easy way to confirm this is to write a script that accesses objects from another thread without a Dispatcher.InvokeAsync.
          https://ninjatrader.com/support/help...-threading.htm

          Further, below is a link to a forum post on demonstrating NinjaTrader is able to take advantage of a 16 logical processor system.
          https://ninjatrader.com/support/foru...20#post1035120


          Specific to optimizations, taking advantage of IsInstantiatedOnEachOptimizationIteration being set to false, using class level variables, and only instantiating objects in State.DataLoaded and only when absolutely necessary, resetting instead of re-instantiating where possible, can reduce the memory and drive seek and transfer time and allow the CPU to do more work at any given moment.

          Below are some threads discussing IsInstantiatedOnEachOptimizationIteration.
          https://ninjatrader.com/support/foru...39#post1069339
          https://ninjatrader.com/support/foru...036#post838036
          https://ninjatrader.com/support/help..._practices.htm

          Also, below is a link to a forum post that discusses why NinjaTrader will still start new threads for a core, even when no optimization iteration will be run in that thread.
          https://ninjatrader.com/support/foru...57#post1045557
          https://ninjatrader.com/support/foru...288#post726288

          There is a processor count as NinjaTrader starts up (visible in the trace) you can use to confirm the amount of logic processors the operating system makes available for NinjaTrader to utilize.
          There are the lines from my computer.
          2020-11-17 07:30:55:600 Processors=8
          2020-11-17 07:30:55:600 ThreadPool: minWorkerThreads=8 maxWorkerThreads=32767 minCompletionPortThreads=8 maxCompletionPortThreads=1000

          This processor count is used for pre-creating threads that will be used for optimizations, as this allows NinjaTrader's IsInstantiatedOnEachOptimizationIteration as false to re-use iterations without demanding more memory.

          As far as the total utilization, there may be other factors about the machine that may come into play such as the windows power settings. NinjaTrader is designed to use all logical processors available; specifically, it creates new threads and thread pools for the operating system to manage and does not limit or throttle the CPU core usage.
          Chelsea B.NinjaTrader Customer Service

          Comment


            #6
            Chelsea,

            Thanks for the list above. I went thru my code and confirmed that I had followed NT Best Practices. I have.
            Then I started to think if I had changed any Windows 10 Pro settings that might have created the problem. Then I remembered that a few weeks ago I Updated the Bios. So I got into the Bios and clicked the 'Set Defaults' option . Rebooted and started NT8. Strategy Analyzer worked has expected, single instrument optimization firing on all cores and CPU Utilization running 70's and low 80'.s. Batch Test Optimization ( SP 500 | Select All) was firing all cores with CPU Utilization in the high 40's.

            So I am guessing that some setting while Flashing the Bios did not catch. Or when I was looking around the Bios screens I accidentally changed a setting. This was probably causing Windows issues and it flowed down to NT8 while the computer was under heavy load. I have no idea what setting(s) was the culprit.

            If anyone following this thread is experiencing the same issues you can not rule out the Bios as a potential problem. Either from Flashing the Bios, Over Clocking or changing any Bios Settings.

            Comment

            Latest Posts

            Collapse

            Topics Statistics Last Post
            Started by LeroyO, Today, 04:09 PM
            0 responses
            1 view
            0 likes
            Last Post LeroyO
            by LeroyO
             
            Started by sidlercom80, Today, 04:57 AM
            2 responses
            29 views
            0 likes
            Last Post NinjaTrader_Jim  
            Started by Tartagliagroup, Today, 02:11 PM
            1 response
            9 views
            0 likes
            Last Post NinjaTrader_Jesse  
            Started by rickyblah12, Today, 01:33 PM
            1 response
            12 views
            0 likes
            Last Post NinjaTrader_BrandonH  
            Started by rcloutier, Today, 01:17 PM
            4 responses
            14 views
            0 likes
            Last Post rcloutier  
            Working...
            X