Unusable During Tick Backtest

[tmfdouglas]

Hi,

When running a backtest using tick data the rest of the platform becomes unusable, e.g. when trying to view a chart it takes 10's of seconds for it to load, when trying to navigate through menus they again take 10's of seconds if not longer. I'm assuming this is due to threading.

This can be tested by running a backtest from 1st Jan 2016 to the current date, 1 tick interval on all futures in the default futures list.

Are there any plans to resolve this?

Thanks

[NinjaTrader_JessicaP]

Hello tmfdouglas,

So we can have a clearer picture of what you are seeing on our end, could you provide a screen shot of your System control panel window?

To send a screenshot with Windows 7 or newer I would recommend using Window's Snipping Tool.
Click here for instructions
Alternatively to send a screenshot press Alt + PRINT SCREEN to take a screenshot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save as a jpeg file and send the file as an attachment.
Click here for detailed instruction

[tmfdouglas]

Screenshot attached. I've tested on both this computer and my desktop also with an i7 and 32gb ram. The problem has been across all beta versions, not reported it yet as assumed it would be fixed as the beta progressed.

If you run the test I described on your own machine I expect you'll encounter the same problem.

Also when running the test the CPU / memory on the computer is no where near maxed out.

Thanks

[NinjaTrader_JessicaP]

Thank you for providing this information. I believe we have enough to investigate with on our end. I will return to this thread when we have more information. Thank you for helping to improve NinjaTrader 8.

[NinjaTrader_Brett]

What you see in product is actually expected. Since when running an optimization we were carefull to make enhancements to really use all system resources to get all performance possible from a PC. We made significant improvements by rearchitecture how NT8 handles optimizations over NT7. As a result we now can efficiently use all PC resources. Which doesn't leave many resources left over for doing other tasks and as a result you'll experience the entire PC is out of processing power while running a heavy optimization since all of it is being used by the optimization.

We could try something like 'artificially limit' the throughput of the optimizations in order to leave some PC resources for other tasks but have optimizations complete slower but wanted to wait for feedback from users before taking any other steps.

[tmfdouglas]

Jessica, Brett, thanks for the quick responses.

It's great how NT uses all the available resources, after further testing today it appears that if the data is already downloaded and available NT proceeds through the backtest very quickly using the majority of the available cores and whilst the platform does hang it is more intermittent.

Although... if the data isn't available and has to be downloaded the platform hangs and is unusable for the majority of the backtest (and takes significantly longer although that is as expected). During this time the majority of the cpu cores are available and memory consumption is far below the maximum available.

Happy to test further if needed.

[tmfdouglas]

Also this is during a backtest type of "standard", I haven't tried with it set to optimisation.

Thanks

[NinjaTrader_Brett]

This case is a little different then an optimization so thanks for pointing that out. However its similar in that there are multiple backtests going. Of course if data has to be downloaded the CPU / memory wont be a bottleneck and NT must wait for data to be downloaded. However if your saying that NT is more unresponsive while downloading data then I will need to investigate that.

Who is your data feed provider so I could test on my side?

Thank You.

[tmfdouglas]

It's NinjaTrader Continuum. My data subscription is with a live account using the CME bundle.

Thanks

[NinjaTrader_JessicaP]

Thank you for this additional information, tmfdouglas. We will get back to you when we have more information.

Please keep an eye on the NinjaTrader 8 Release Notes page for updates and bugfixes.

http://ninjatrader.com/support/helpGuides/nt8/en-us/release_notes.htm

[tmfdouglas]

Any update?

Thanks

[NinjaTrader_JessicaP]

One way to artificially limit NinjaTrader's resource usage, similar to what Brett suggested earlier, is to right-click your NinjaTrader 8 64 bit shortcut, and change its target to the following :

C:\Windows\System32\cmd.exe /c "start /low C:\PROGRA~2\NINJAT~1\bin64\NinjaTrader.exe"

You may need to adjust the short names if that fails to start Ninja. You can learn a short name (PROGRA~2 and NINJAT~1) with this procedure

• win + r, type in "cmd" without quotes, enter
• cd %ProgramFiles%\..
• dir /x
  • Record the short names of (both of) your Program Files folder(s)
• cd "Program Files"
• dir /x
  • Record the short names of any NinjaTrader folders
• cd %ProgramFiles(x86)%
  • if the previous command succeeds
    dir /x
    • Record the short names of any NinjaTrader folders

The bolded part of the target I sent allows you to change the process priority for NinjaTrader.

[NinjaTrader_Brett]

Also still working on this internally to give you the answer of what is expected by default.

[NinjaTrader_Brett]

Hello,

Wanted to update on you we found the culprit. Had to do with the fact that we were loading daily data to calculate offsets for the entire offset range and not just the year back that you needed. This drastically slowed down data download. Next release you should see improvement here. Thanks for reporting.

[tmfdouglas]

Great, thanks for investigating so thoroughly. I'll look forward to the next release.