NT8 Freezing During High Volume Periods

[markbb10]

Hello All,

Was hoping to get some input on the following issue. During the last few days my NT8 has been pretty much frozen/lagging during the first half of the day due to the high volatility and volumes.
I have had this issue before and was hoping for some ideas on the best way to deal with this problem or is it just a limitation in the architecture design of NT8. The freezing is due to screen
rendering where many times the charts are lagging 15 - 30 minutes behind the market. The DOM is always accurate reflecting correct prices, it is only the charts that lag.

Some background

My computer resources are never really taxed during these episodes. I am never using more then 50% of my available memory or CPU.

I only trade one instrument which is the NQ. so the vast majority of my screens (charts) are for a single instrument I use tabs for my charts and usually have 4 charts active and
visable for the NQ and 2 for the ES.

I trade orderflow so most of my charts are tick based but I am careful and other then footprint charts set updates to occur on Bar Close for the majority of my indicators.
The problem seems to be exasperated with the NQ, if I use my same workspace, charts and indicators but monitor the ES the problem usually does not surface

I have run the Ninjascript Utilization Monitor throughout the day and vast majority of time is being spent on the NT Orderflow Footprint and NT Orderflow Volume Profile
which I would expect.

Being a user of Ninja for many years I routinely rebuild my entire environment from scratch every 60 days, I have found this seems to help prevent lots of mysterious problems that seem
to happen with a install and DB that has been around for some time. I clear all my caches and logs every morning as part of my startup process.

I am hoping to get some ideas around the architecture of the system. Is there anything I can do from a setup or architecture perspective to address this. Am I using a worst case environment (tick charts, one instrument, order flow, etc) and this is what I should expect.? Do I need multiple instances of Ninja Trader (licenses) to distribute the work?

Any input or ideas would be appreciated.

Thanks,
Mark

[pjsmith]

very common. I and many others have had the same issues. Still exists a bit for me, but what I found helped a lot, is reducing the number of bars on the screen. It seems a little counter intuitive. Although I have charts with tick series for volume profile etc, it's really only the low tick type charts that mainly cause the lag. Reduce number of bars on any short term charts with tick series to a minimum. Indicators can cause issues, but on the whole, NT cannot cope in most cases in volatility liek recently with just a few charts for the same instrument open, and a few volume profiles. I recently thew a new, amazingly powerful PC at it too, but with minimal inpact. NT I believe runs all the same instrument through the same thread, so if you have mutiple charts for same symbol, that will slow you down too. Other thing I found helped. I put some of my long term charts in 'tabs'. When a tab is hidden, many of the indicators can pause and save resources whilst you are not actually looking at that chart.

Hope it helps, but I think you will not get a 100% solution. Unless you are running bare chart only and nothing else, I have never seen NT cope fully my side (and I know a lot of others that experience same).

[markbb10]

very common. I and many others have had the same issues. Still exists a bit for me, but what I found helped a lot, is reducing the number of bars on the screen. It seems a little counter intuitive. Although I have charts with tick series for volume profile etc, it's really only the low tick type charts that mainly cause the lag. Reduce number of bars on any short term charts with tick series to a minimum. Indicators can cause issues, but on the whole, NT cannot cope in most cases in volatility liek recently with just a few charts for the same instrument open, and a few volume profiles. I recently thew a new, amazingly powerful PC at it too, but with minimal inpact. NT I believe runs all the same instrument through the same thread, so if you have mutiple charts for same symbol, that will slow you down too. Other thing I found helped. I put some of my long term charts in 'tabs'. When a tab is hidden, many of the indicators can pause and save resources whilst you are not actually looking at that chart.

Thanks PJ, can you expand on what you mean by reducing the number of bars on the screen, do you actually mean just have less bars visible ? My lowest value tick chart is a 1000 tick. If they truly architected this to have a single thread handle a single instrument that would explain alot. So you could have a top of the line chip that basically only makes use of a single thread of processing power. Hopefully NT support will read this and confirm, that would be a major architecture flaw.

I do use tabs for my charts so that is already in place.

Mark

[NinjaTrader_ChrisSch]

Hello markbb10,

Thank you for your note.

I would like to analyze your diagnostics files for more information.

Please follow the steps below to manually attach your log and trace files to an email to platformsupport[AT]ninjatrader[DOT]com with "ATTN CHRIS S 2232467" in the subject line so I may investigate this matter further.

• Open your NinjaTrader folder under Documents.
• Right click on the 'log' and 'trace' folders and select Send To> Compressed (zipped) Folder.
• Send the 2 compressed folders as attachments to this email.
• Once complete, you can delete these compressed folders.

I look forward to your reply.

[pjsmith]

Thanks PJ, can you expand on what you mean by reducing the number of bars on the screen, do you actually mean just have less bars visible ? My lowest value tick chart is a 1000 tick. If they truly architected this to have a single thread handle a single instrument that would explain alot. So you could have a top of the line chip that basically only makes use of a single thread of processing power. Hopefully NT support will read this and confirm, that would be a major architecture flaw.

I do use tabs for my charts so that is already in place.

Mark

Hi Mark - I mean, for example- I have several charts for same instrument. Long, medium, and short term for example. All have tick series on. The long term charts however, becuase of their time frame, have less bars, and I find it is the short term chart, 1 or same day, that actually will kill your performances. If this is several days for example, NT8 deis. Set is to 0 days to just get todays session, and it's much better. Better, if 'Bars' count work work properly (it does not, IMO), I would just have 2000 bars or so one my entry chart. It's not visible bars, it's total bars on char (ignoring timeframe). For me, at least, this is without a doubt, a factor. btw - NT are well aware of the architecture and it's flaws (if you want to call them that - I would). Many other threads over _years_ discussing it. When choosing a PC, depending on how you are setup, more clock speed might be better than more threads, for NT, at least...

[markbb10]

Thanks PJ - So I have several tick charts (same instrument) that are loading 45 days of data. I use this amount because I am calculating support and resistance zones and need to go that far
back because of the volatility of price movement . As I said I trade a 1500 tick chart and a unirenko chart and monitor several others 15M, 60M as well as a 1000 tick footprint all on the NQ. So if I am reading your note correctly I would be up a creek true ?

[pjsmith]

LOL. Yeah, that was me a long time ago too. You've just got to try your best to cut it all down and do it in ways that have minimal effect as possible on Ninja. I now use minute bars for volume on >30 day charts for example, but zoom in to tick level on <30 days. I only plot current session for very short term charts. I try to sometimes combine stuff. For example, chart that has 3 indicators that use tick series - If you can, remove the tick series and turn on tick replay instead 9tick replay is normally evil, but it may have uses like this). That way, all 3 indicators use same tick series (I'm not sure if my logic here is valid as I have no internal knowledge of nt8, but it does seem to be relevant). I've spent literally 100's of hours optimizing for performance workarounds. I pity the guy who uses off the shelf stuff and does OF at this level, in real-time on this platform, seriously...

[markbb10]

Thanks again, I am going to have to sit back and rethink this. With the processing power available today we should not be having these issues. Thanks PJ, I may follow up with you more as I delve deeper into this. I see relative volume today is up at like 280% so this is going to be another tough day navigating ninja.

[pjsmith]

Thanks again, I am going to have to sit back and rethink this. With the processing power available today we should not be having these issues. Thanks PJ, I may follow up with you more as I delve deeper into this. I see relative volume today is up at like 280% so this is going to be another tough day navigating ninja.

fwiw, with my new PC, CPU usage as a whole never really goes over about 10%. Charting live instruments like NQ that are jumpy in recent volatility, it still lags (though nowhere near as bad as it used to, given all the changes over the months/years). There were times when I would literally be 30 minutes behind the market in the old days!. Pushing those ticks through each individual gate in turn on a single thread before processing the next tick. It's just too much for what most people have running.

[Lancer]

Script to monitor chart lag:
ChartLagTimeV3.zip

[markbb10]

Thanks Lancer, I will check that out. My charts will probably be rendering with a 30 min lag today.

[pjsmith]

Thank you to Lancer for posting this simple, but very useful script. In fact, I have been running it a few days now, and I'm pretty shocked enough by what I see to actually stop trading anywhere near as much... Seriously. If you are scalping, short term etc., you need to check this...

Couple things to note - Your local clock is likely off by a bit, but will at least remain constant. If on windows etc., sync it using the OS option to the online clock to try and get it as good as possible before you test, but regardless - You'll get a consistent result, even if your clock is out. I think the main thing to watch for is the variations.

What I have noticed (and bear in mind I am wrong side of the pond for US indexes), my latency is around 400ms. This morning, for some reason, 800ms. I am an IT network guy that used to run 15 global sites. I can tell you, that is truly appalling. I'm expecting, even with the transatlantic latency, somewhere around 200ms. Over last few days, in time of brief volatility, it goes from 1-3 seconds. if you were not monitoring this, you'd never actually know and that sort of lag can be extremely hazardous to your wealth (if you are short term trading). During the close, last 10 minutes or so, and extreme volatility, I've seen it at 9 seconds or more. I'm running it on this beast (attached) - All solid state drives etc., Minimal indicators etc.,Never even breaks a sweat. All code as optimised as I can get it. Clearly, it is nowhere near good enough. I've run the utilisation monitor, and it shows the candle stick bar type uses as much resources as anything else on my system, so I don't think there is any pointing fingers at my code. In fact, as I mentioned, I have done extensive optimisations, even to the fact of removing the candles bar type and replacing with a type that does less, where I don't need bars on charts. For info, I have 6 screens and about 15 charts visible at one time, a few have more than one data series on them.

Lancer - Thanks again for opening my eyes. I am truly shocked by what I have found.I owe you a beer

Please, anyone trying this, post your results here, so we have a baseline of typical users, and approx where you are on the globe, so we know how much of the latency to apply to flight time. Might also be good to state your data provider, as I am sure that will make a difference in places,. Mine is currently NT brokerage, using continuum. Cheers all.

[pjsmith]

Thought I'd add to this. Been out all morning, so left utilization monitor running after a _clean_ restart of PC and ninja.

Top 2 resource hogs are both NT supplied inidies. Order flow market depth and volumetric bars. I have ONE 1000 volume NQ volumetric chart for 3 days. TWO order flow market depth 1 on nq, 1 on es, with historical plotting only. Now, tell me it's my indicators screwing up ninja and making it slow!

[NinjaTrader_PatrickG]

Items being in the NinjaScript Utilization Windows are not inherently a bad thing. NinjaTrader must necessarily take processing time to perform tasks. Additionally, this window is cumulative - the longer you leave it open, the more time will be added to the total processing time for an item. This tool is most useful in short duration (like 5 minutes) to determine if any one item in your workspaces is taking considerably more processing time than another item. How long was the tool open?

With this in mind, can you tell me 1) what you were experiencing and 2) what you were trying to determine with your test?

[pjsmith]

Patrick - Sorry, but been through it all before. Usually, the response is, remove all your indicators, start with a clean workspace, etc., see if issue persists. If you add back indicators one at a time, etc., etc. I just cannot do it (again). I've done it many times since version 7 over years! I was just demonstrating, and here is one running from just a few minutes whilst I type this, that the fairly blanket response does not cut it. If NT supplied indicators in _very_ sparse usage are taking more resource time than anything else on the platform, why does it lag so bad in times of volatility? Symptoms are all covered above, as are expectations, in many posts, and in many other user posts in this forum. Sorry if this sounds like I am venting (I am a bit), but you could call me a frustrated 'semi professional' user of NT8.