Unlock limited CPU usage on NT8

[RaddiFX]

Hey there,

I was wondering if there were any way to increase the performance on NT8. I have an addon that pairs with a strategy that I've tuned to be as quick as possible at this point. I'm still seeing a bottleneck occur from the CPU and NT8 stops responding while processing this. My problem is, it's bottlenecking @ ~45/35%(total/utilized by NT8) CPU usage on all cores, memory goes almost unchanged, network only bottlenecks if it needs to download entirely new ticker data, and disk never bottlenecks even though it will sometimes reach 100MB/s IOps at some points.

For clarity, I did a bench test, and I'm able to utilize 100% of my CPU.

So this discovery led me back to my initial question, is there a way of "unlocking" that cap, or improving the top end performance of NT8?

[RaddiFX]

I just did a test using the strategy analyzer on 1000 instruments to make sure it took a decent amount of time and ran into the same ~40-50% limit on CPU usage.

[RaddiFX]

I've also, through testing, determined that this only happens when I'm connected to Kinetick. If I'm only connected to IBKR then it runs at 100% CPU. However, there is no spike in network usage.

[NinjaTrader_Jason]

Hello RaddiFX,

Please see the performance tips and suggestions at the link below.

http://ninjatrader.com/support/helpGuides/nt8/en-us/performance_tips2.htm

[RaddiFX]

Thanks for the weekend reply Jason!

I'm following as many of those as possible. and the one that seems to be causing this is:

Different connectivity providers (market data vendors and broker technologies) that NinjaTrader supports vary in their level of real-time data service.
For example, providers who deliver unfiltered tick data (submit all market ticks) will impose heavier processing load than a vendor that provides throttled market data.

I can't limit it any further as my strategy is running at the tick level so it will be intensive. However, I'm only getting these short periods of "not responding" occuring when starting a strategy or running backtests while connected to Kinetick. It seems that the attempt to download historical data, most of which I already have in my database, is what's possibly causing the issue?

I've already slimmed down and sped up my code as much as possible, it can process over 100 tickers/sec(when not connected) and I can do tick level market replay on the strategy with over 800 without overloading. I've also created a custom tick type so that days to load is always 1 no matter what day of the week it is(2 on Sunday, 3 on Monday, 1 every other day) to limit as much of this processing required.

At this point, I need to figure out if there is something I can do about Kinetick load times.

Or, is there some way for me to preload the data?
Or create a service that is constantly subscribing and unsubscribing from tickers as it updates my database and my primary strategy just not trigger kinetick updates?

[RaddiFX]

Update: I got the free trial of BarChart data feed and they're ~1.5x as long to load.

Which leads me to the conclusion that the bottleneck itself is in the Ninjatrader core code. And how it loads/requests the data when starting up a strategy.

Some additional testing: For both BarChart, and Kinetick, extending the days loaded from 1 to 5 is ~15-20% increase in load time. So load times do impact it but not linearly.

[NinjaTrader_Jason]

If you download the historical tick data you need before enabling the strategy, does the freezing behavior persists? You can download historical data at Control Center-->Tools-->Historical Data-->Load-tab-->Download.

If you apply the 'Sample MA crossover' strategy instead of your strategy, do you experience unresponsiveness?

[RaddiFX]

Yes, and yes.

This really only becomes an issue during news release times, 7, 7:30, 8, 8:30, 9, 9:30 AM. During those times sometimes 80+ news articles can come out with important sentiment and will be loaded into NT. The problem exists in that it's either trying to download, or just verifying that it has all of the data for the required load. I pulled a wireshark of one of these periods and it's non stop communicating, but the packets are only a few bytes at a time. ~100Kbps.

This could be resolved if it could enable the strategy with known data and "backfill" what it does download from Kinetick. There are probably a lot more ways to fix it but I'd need to know the steps taken to enable a strategy in the first place.

[NinjaTrader_Jason]

Can you please send me a screenshot of the settings you are using for the 'Sample MA crossover' strategy when you experience freezing behavior upon enabling this strategy. Did you already had your strategy running when you applied the 'Sample MA crossover' strategy?

When you refer to "news articles", are you referring to articles in the News window? Does your strategy somehow uses these news articles?

You can enable strategies in charts and the Control Center-->Strategies-tab

Chart
https://ninjatrader.com/support/help...g_with_automat ed_strateg.htm

Strategies-tab

https://ninjatrader.com/support/helpGuides/nt8/NT%20HelpGuide%20English.html?strategies_tab.htm

[RaddiFX]

Default settings, I'll upload a video once the market closes. It actually takes longer for the 'Sample MA crossover' to process than my strategy because it processes on historical data(when loading) which mine does not.

Because there are over 10k instruments in the stock market to keep track of, and I can only track 100-200 at a time through Kinetick, I created a service to import a list of tickers and open and enable strategies on the fly based on news I track. This issue is in regards to that, but it happens with or without the service. The best way to emulate it is to select 20 disabled strategies in the strategies tab and enable them all at once, this lag occurs. Because that's literally all that it's doing.

[RaddiFX]

Sorry, I'm going to need to put troubleshooting this on hold. There was a larger issue that occurred today that I think might be related to a few missed trades recently I need to address.

[RaddiFX]

Just a note while I'm looking for info on something else. I took a look at the trace logs, and it seems to make multiple requests for data, which is because I'm using multiple bar types. What I find unintuitive, and excessive, is that it will request data whether the database is up to date or not. Which means another solid solution to this issue would be for the NT to check the database first, and only request the missing data.

2021-10-11 21:16:22:685 Cbi.Instrument.RequestBars (to Provider): instrument='APPL' from='9/27/2021 12:00:00 AM' to='10/11/2021 12:00:00 AM' period='1 Minute'
2021-10-11 21:16:22:694 Cbi.Instrument.RequestBars (to Provider): instrument='APPL' from='9/27/2021 12:00:00 AM' to='10/11/2021 11:59:59 PM' period='1 Tick'
2021-10-11 21:17:15:320 Cbi.Instrument.RequestBars (to Provider): instrument='APPL' from='9/27/2021 12:00:00 AM' to='10/11/2021 12:00:00 AM' period='1 Minute'
2021-10-11 21:17:15:320 Cbi.Instrument.RequestBars (to Provider): instrument='APPL' from='9/27/2021 12:00:00 AM' to='10/11/2021 11:59:59 PM' period='1 Tick'
2021-10-11 21:17:24:851 Cbi.Instrument.RequestBars (to Provider): instrument='APPL' from='9/27/2021 12:00:00 AM' to='10/11/2021 12:00:00 AM' period='1 Minute'
2021-10-11 21:17:24:851 Cbi.Instrument.RequestBars (to Provider): instrument='APPL' from='9/27/2021 12:00:00 AM' to='10/11/2021 11:59:59 PM' period='1 Tick'

[NinjaTrader_Jason]

Please see the link below that explains when NinjaTrader loads historical data.
https://ninjatrader.com/support/help...g_historical_d ata.htm

If you try the 2 ways described under "The NinjaTrader memory and speeding up the loading of data", does the freezing behavior persist?

[RaddiFX]

Both of those suggestions remove the delay. However, I do not preemptively know which instrument I will need to load from one moment to the next and am limited to 100 tickers. I will usually load/unload around 100 every 30-45 minutes during the 8-10AM time frame.

[RaddiFX]

I'm back to this being my primary issue haha. So I've done some testing and my results seem accurate. The issue isn't the datafeed itself, it's the way in which ninjatrader starts the strategy.

Is there a way to change the way that ninjatrader requests the data? or a setting I could use so that it doesn't request historical data when starting a strategy.

The reason I ask that is because I can make a service that preemptively cycles through and requests data for all tickers individually to keep them up to date within a 10 minute delay. but for that to be worth anything, any of the following would need to be possible.

• Ninjatrader checks its local DB and only requests missing data. Rather than all data from the period loaded.(best and most logical by far)
• Strategy setting to have NT not request new/updated data.
• Global setting to not request new/updated data and I can update the database through manual requests.
• The code following NinjaTrader.Gui.NinjaScript.StrategiesGrid.Strateg yEnable to be overridable so that it doesn't call to request historical data in any of the above ways.

I'd actually be able to implement the last one myself if you could provide me with the code that the following block calls

foreach(var s in NinjaTrader.Gui.NinjaScript.StrategiesGrid.AvailableStrategies){
//do some stuff to make the strategy object usable
try{
    s.Instrument=ins;
    NinjaTrader.Gui.NinjaScript.StrategiesGrid sg=new NinjaTrader.Gui.NinjaScript.StrategiesGrid();

    System.Reflection.MethodInfo dynMethod = sg.GetType().GetMethod("StrategyAdd",
    System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static);
    if (dynMethod!=null)
    dynMethod.Invoke(sg, new object[] { s });

[COLOR=#c0392b]    dynMethod = sg.GetType().GetMethod("StrategyEnable",
    System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static);
    dynMethod.Invoke(sg, new object[] { s, null, null });[/COLOR]

}
catch(Exception ex){NinjaTrader.Code.Output.Process("error "+ex.Message, PrintTo.OutputTab1);}

Whatever code follows the red section would need to be known before I'd be able to override it.