Unusable During Backtest

[tmfdouglas]

Sure I've reported it before but I've emailed evidence of NinjaTrader 8 being unusable during a backtest, this is not due to lack of system resources (windows performance monitor is captured in the video). I can't upload videos to the forum, hence the email.

Also the first time I ran this backtest I exited NinjaTrader, but it did not exit the NinjaTrader processes correctly which we're still running. The backtest in both cases was the sample MA crossover so there is no custom code.

At least on the positive side NinjaTrader did eventually complete the backtest without crashing (the first time it probably would have too but I killed the process as it became unresponsive).

Backtest was across all FXCM instruments, from 01/01/2016 - 07/05/2016, fast 10, slow 20, high fill resolution 1 tick, price based on last, standard backtest.

Log and trace files have been sent (with the video attached).

Please provide:

Likelihood of NinjaTrader fixing this?
Expected ETA?

[NinjaTrader_PatrickH]

Hello tmfdouglas,

Thank you for providing this report and the supplemental information via email.

In short, this behavior is expected. When running an optimization in NinjaTrader 8, the platform will distribute the load to all available cores on your machine and attempt to consume all available resource on each core. This was done to maximize performance at the expense of application responsiveness. You can abort the process if you want control of the application back. Since NinjaTrader 8 was released in beta, we have been monitoring for feedback in case demand warranted a change in approach. In your case, a multi-instrument backtest is no different than an optimization in the approach to maximize CPU resources across all cores.

I will submit a feature request for future consideration to have the application refrain from using all available resources.

[tmfdouglas]

Thanks for the detailed response Patrick but what you describe is not what's happening and the video I submitted clearly shows that. When running the backtest cpu averages around 10 - 15%, it is not using all available cores (of which there are 16, if it used all of them that would be a good feature), this is shown in windows performance monitor at the right of the video submitted.

As the video evidence proves it is not caused by the situation you describe what else could be causing this?

[NinjaTrader_PatrickH]

Hello tmfdouglas,

Thank you for your patience.

We are looking into this further on our end. I will follow up with you when I have any details on this matter.

[NinjaTrader_Brett]

tmfdouglas,

I'm trying to isolate your scenario so I understand what is going there.There are a few things, going on that I want to check in.

First of all, with a basket backtest we need to build bars to which run our backtest on. In a basket backtest this is likely the bottleneck since we need to download data from the provider and build the bars before we can start a backtest. There is a setting on the strategy analyzer (right click > properties) Use Local Data Only. Which when enabled will only use locally stored data instead of getting data from provider. When using NinjaTrader historical data servers for data we only fill 8 concurrent requests at a time from the server so thats as fast as it will go.

However that does not in and of it self does not explain the UI unresponsiveness you saw. I setup a test with SampleMACrossover on my side by creating an instrument list with all forex pairs and all CFD pairs about 88 instruments. Triggered a basket backtest with that 'simple' strategy and saw no problems.

Hence I'm missing something, either something specific about the strategy or something specific about some options your using. OR -> There is something different with your server style PC with the large amount of cores which is causing unexpected behavior and the bug could be there.

We should start with the strategy though, can you explain a bit more about your strategy. Are you adding in any secondary data series or have any code which could be 'long running' that I could write a sample with on my side to try to reproduce?

Thank You.

[tmfdouglas]

Hi,

It's just the standard sample strategy.

In your tests did you set the order fill processing to high 1 tick and backtest from the start of 2016?

[NinjaTrader_Brett]

I did SampleMACrossOver, I also did 150 tick bars from Jan 2016 and also set high tik resolution. I recorded a video of my second run through. I already has some of the data so it only took me about 4 minutes on 88 instruments to basket backtest that. No lock ups, and CPU is @ 100% on all cores when its cranking through backtests and dips down as you see network activity come up when downloading data to build the complete bars.

At about the 2 min mark I started working with menus to make sure responsive:

https://www.screencast.com/t/bk16F9x1Vm

I'm missing something from your test to mine, this is on a server box yes? Willing to let me remote on and take a look?

[NinjaTrader_Brett]

Disregard my last post, turns out I wasn't fully connected from an earlier test so I wasn't downloading a years worth of data. I will return with an analysis.

[NinjaTrader_Brett]

Alright, a single backtest of 1.5 years back of tick data generating 1 tick bars for FXCM historical data has the following resource requirements for a single iteration:

* 4gb of memory
* Writes 114MB worth of cache data (Highly Compressed)
* Writes 114MB worth of tick underlying data.(Highly Compressed)
* Downloads about 114MB worth of actual data.(Highly Compressed)

Stacking that up with a basket backtest with 80+ instruments to backtest and resources we're talking resource requirements for such a basket backtest are extremely high. I couldn't even generally pull off the curb with by 16 gigs of RAM available. I saw unresponsiveness for different reasons which was I simply was out of local resources.

Your scenario is a bit different though, you have a huge machine from the looks of your video with 16/32 cores and 126GB of RAM if I recall correctly from the video. In that case the I/O operations are on generating the cache and downloaded the data which comes 8 at a time from the HDS are likely the bottleneck for your case which is why you don't see CPU starvation.

However one last outlier remains unanswered which is the fact that you had plenty CPU/RAM resources yet you still saw UI responsiveness degradation. We acknowledged that and will create a JIRA of which to research the case as we get resources. I wont get immediately back but will return shortly. The JIRA for that case is: NTEIGHT-11685