NT8 - Strategies - Slow loading - Slow to load - A suggestion...

[bertochi]

Hello,
The strategy loading time appears to depend on the number of strategies running, number of time-frames traded inside those strategies and the days to load parameter.
There have been quite a few old posts related to NT8 slow loading.
I would like to share my experience and maybe a suggestion to the dev team.
Ok. I have checked and tested this extensively and found out that even when your historical database and cache "data" is updated up to the last minute, when you restart NT, the cache is fully restored, all files overwritten, and if you have many strategies, with different timeframes and many days to load, this process takes an unnecessary long time, a couple of minutes, when ideally the cache is already there and does not need to be fully rebuilt. This is a huge performance issue and maybe something to consider by the dev team.
Thanks

[NinjaTrader_Kate]

Hello bertochi,

Thank you for your reply.

Each of the items you mentioned does impact performance, that is true.

However, it would be expected that if you have many strategies that each are pulling in multiple time frames of data, that this would take some time to load and for the strategies to all process. This would me most noticeable on Tick Based charts as these use the most granular data. Rendering is another item that can impact performance on a restart of NinjaTrader - if your strategies draw lots of objects on the chart that can definitely impact how quickly the strategies load.

Here's what happens when a chart is opened or a bars request is made by reloading historical data or from a strategy:

1. NinjaTrader starts a realtime subscription to market data with the provider and starts caching realtime ticks in memory
2. NinjaTrader checks if you already have the necessary bars in the cache to fulfill the bars request for the chart
   1. If so, go to step 5
3. If not, NinjaTrader checks if you have the necessary data in the repository to fulfill the bars request
   1. If so, go to step 4
4. If not, NinjaTrader sends out a bars request to the provider/HDS
   1. When this request is returned from the provider/HDS:
      1. The data is saved to the repository
      2. The data is processed through the selected BarsType code to build bars
      3. Then the bars are saved into the cache except in the following circumstances:
         1. Tick Replay
         2. Volumetric Bars Type
         3. BarsBack request
5. When the bars request has been fulfilled, NinjaTrader needs to merge the historical bars with the realtime subscription
   1. By starting the subscription before the bars request in step 1, we try to ensure there is some overlap of data so we can align the two data sources.

So, it's not "fully rebuilding" the cache, it reloads the cached data and then adds to it.

If that's not the behavior you're seeing, I'd be happy to take a look at a test script you've used to test and steps to duplicate what you're seeing.

Thanks in advance; I look forward to assisting you further.

[bertochi]

Thanks for your reply Kate.
I only use strategies, I do not use charts or any other feature from NT.
The thing that got me puzzled is that strategies load instantly when the market is closed. So the loading time is all due to waiting times of the requested bars to and from the price server provider. When using a strategy with only one timeframe, loading time could be as fast as 30 secs. When using a strategy with 6 timeframes loading time could be as slow as 10 minutes. I checked the trace file and NT is indeed requesting bars for each of the timeframes and it is really painfully slow. CPU hardly goes above 20% so that is not the issue, which appears to be the requested real time data, which as I said on my first post, should not take long if you had it all until a minute ago when you restarted NT. I noticed this and I thought I should bring it to you as you know how NT works much better than me. I would have thought there was some room for performance improvement here.
If I did not need to restart NT in the middle of the day, this would not be a problem, but since NT still has many bugs when used with TDA, it requires many restarts within the day, and the loading times are unbearable.
Thank you once again for your quick response.