Make chart refresh rate faster than 250ms possible

[pjsmith]

fwiw - I see those errors in my log too, but they were there long before I tried this indie. Pretty sure this indie is not the cause...

[brucerobinson]

Re: error message previously referred to - checked Logs pre-dating addition of TickRefresh and the error message appears. Hence fair to conclude un-related to TickRefresh and dismiss (checking it out presented a whole new rabbit hole I don't want to go down, seems to suggest to do with SuperDOM ;-()

Re: latest version - thanks, wasn't aware of it. Installed and hence will be in use hereafter.

FYI: I did notice slow Global Crosshairs yesterday at the Open for a while. Same caveats re: any comments as before - I'm not saying that wasn't/wouldn't be so without TickRefresh, (it isn't like Global Crosshairs are not prone to lagging in fast markets, and wasn't a problem at other times, and all go beserk together at the Open. TickRefresh may have made no difference or no perceptible difference to them whatsoever I'm just scrutinising at the moment to an extent not normally the case). Etc, etc, etc. Just feeding back observations as we go.

See what today brings

Kind regards,

[hurleydood]

Thanks for the update Bruce. I have not noticed crosshair slowness with the indicator. From time to time they do randomly stutter and get jumpy even when the market is closed. From what I understand the crosshairs are not even rendered on the chart anymore and are overlayed for the best interactivity, smoothness and run on another thread. I have tested out cursor performance with market replay set to the max and the hardware cursor displayed. The lag is non-existent, see this video https://youtu.be/DynCOqL2p8c

[brucerobinson]

Hello world,
As promised a further update. I believe it is now 3 weeks in.
I have been using TickRefresh in the manner I previously described - on my fast tick charts only on fast-movng markets i.e. judicious use only where I believe Ninja's imposed 250ms is a limiting factor for faithful visualisation of realtime price action in the chart.

Short-form summary - I see NO negative impact resulting from adding TickRefresh. It is always difficult to be sure, with many variables not least market behaviour, but I have made no changes to either my setup nor my trading behaviour since I added the indicator and traded as normal every day through times during which we'd most expect such a tool to have an impact i.e. high activity/volatility/volume etc. In essence I've pretty much tried to forget TickRefresh is there.

I'm confident the log errors I pointed to in an earlier post are not related. Same re: crosshair lag.

So, for me, used in my setup, TickRefresh does what it says on the tin with zero detectable impact upon performance.

For info of those who have interest, want faster refresh and might consider using TickRefresh, and taking into consideration Ninja's statements on this topic regarding imposing the retrograde step of 250ms refresh limit being considered necessary & an acceptable constraint, my conclusion at the moment is that I disagree, vehemently.

This sorry tale has gone on since early NT8 beta (effectively since some users first noticed - because the limitation was perceptible to some users negatively affecting their trading, and Ninja has remained intractable on the issue throughout).

My hardware is not top-end - I have a relatively low spec CPU, sufficient memory, a professional graphics card and relatively lots of screen pixels to render & refresh. TickRefresh charts run as smoothly and synchronously as Hurleydoods video clips. I point this out because my conditions/total system load is comparatively much greater than those examples, for anyone who may be thinking 'that's all well and good in a 'lab-test' setup with nothing else going on, but what happens in a real-world setup?'.
Three weeks 12hrs per day real world setup all market conditions multiple markets concurrently through high volatility & volume, NO feared deadlocks. Trading in those conditions is simply a (cognitively) lighter smoother more flowing experience.
It's had some decent stress-testing.

My thanks to all who have voted for, supported and contributed to this thread and particularly to hurleydood for producing a workaround, sharing its workings and the elegance in the approach which provided me the confidence for baby-steps forward, and making it available to all & pjsmith for his valuable input & persistence with the cause. At times of dispondency it is helpful to find like-minded individuals.

I'll post if I find anything to report. However I cautiously don't expect to based on the above. My expectation is that in 6 months I'll be back reporting 'nothing to report'. And that really will be Hallelujah.

Kind regards,

[markbb10]

I cannot believe I just found this thread. As a user of Ninja since 6.5 I was about at my wits end dealing with lag in NT8. Calls to support always ended with it must be a 3rd party indicator problem. As someone who is a full time trader I MUST have a platform that can handle the market daily and was "this" close to scrapping everything Ninja and look at IRT. I am going to give this a try now and see. Today was another lost day of trading for me due to lag. Thanks to everyone on this thread..

[pjsmith]

I cannot believe I just found this thread. As a user of Ninja since 6.5 I was about at my wits end dealing with lag in NT8. Calls to support always ended with it must be a 3rd party indicator problem. As someone who is a full time trader I MUST have a platform that can handle the market daily and was "this" close to scrapping everything Ninja and look at IRT. I am going to give this a try now and see. Today was another lost day of trading for me due to lag. Thanks to everyone on this thread..

Not sure this will help with the lagging (that is IMO, a design flaw), but it will allow you to see price dancing around much more clearly, if you have the resources to do so. Going off on a tangent. I had lagging issues last few days with the volatility. I completely remove the default 'candlestyle' type today and replaced with my own more efficient one. No problems today! Go figure...

[markbb10]

Interesting PJ. The strange thing I see is I use identical charts for the NQ and ES. A renko chart, a candle stick tick chart (1500 tick) and a PnF chart. On days like today the ES keeps up fine but the NQ charts (all in the same workspace) start to lag incredibly till the whole system just grinds to a halt. My HW resources never seem to be a problem. Looking at my charts right now my ES charts are fine and my NQ charts are lagging now by 22 minutes... Yes 22 MINUTES... Perhaps the refresh is not the issue and I am back at square one.
.

[pjsmith]

Interesting PJ. The strange thing I see is I use identical charts for the NQ and ES. A renko chart, a candle stick tick chart (1500 tick) and a PnF chart. On days like today the ES keeps up fine but the NQ charts (all in the same workspace) start to lag incredibly till the whole system just grinds to a halt. My HW resources never seem to be a problem. Looking at my charts right now my ES charts are fine and my NQ charts are lagging now by 22 minutes... Yes 22 MINUTES... Perhaps the refresh is not the issue and I am back at square one.
.

Not to go too OT in this thread, but yes, seen the same thing many times in the past and suffered horribly. It is my understanding that each tick effectively has to pass though each chart and indicator in order (it is 1 thread per instrument) and be processed before the next one is. That explains why one instrument lags, and one does not. Sometimes, you seem to get unlucky and something else gets assigned to the thread too, making it worse, sometimes better! For some reason, thin liquidity, high volatility = Ninjatrader death. I have a really beefy platform and yes, I have custom indicators I wrote, and I've had a devil of a job getting it to run nearly as well as I need it over the years. Lots of compromises over what I want, vs what ninjatrader can handle. I've just spent stupid money getting a newer more powerful box (than the 4.3Ghz 8 core 32Gb machine I have now!), even though my old box NT usage never exceeds about 25% cpu usage. This is because the single threads ninjatrader uses are maxed out. I like to think of them as 'gates'. Each tick has to pass though each gate in order before the next tick is processed. Reduce the amount of gates, improve NT performance. Also, I found that if you are using tick bars (as I do), it is the worst... My 100 tick entry charts for instance. 3 days of bars = death. Have to use the custom day setting to set it to today's range only, and even then, by late afternoon, it begins to lag. It sucks, but Ninjatrader seems to be the best of the bad solutions. several months work at least for me to move away with the code I have. This indicator here though did make a huge difference to the platform and swung me in favour of keeping it until I have a better solution. Until this, I was about to bite the bullet and migrate to something else I have in test.

So, in summary - Reduce tick data as much as you can. Reduce number of bars (most important I think rather than days), reduce number of 'oneachtick' indicators. Even 'onpricechange', set to them to recalculate only every 500ms, or whatever, if your own code. Reduce number of charts on same instrument if possible. All compromises, but you need to do this to get anywhere near the performance required in my experience.

Friday rant over. Hope this helps! I've been using this platform since 7, every day, for probably 14 hours a day. I hate it.... and love it... I just wish NT would concentrate on stability and performance, rather than new features that are almost always broken. I found major bugs in the last release already and had to downgrade 2 days later. Lost a whole day through it.

All just my understanding and experience. I am sure I am the bane of NT support, but I am a real trader who trades full time for a living. I don't think NT staff/testers do.

Anyhoo - Have a great weekend!

[NinjaTrader_PatrickG]

pjsmith, that is a good general explanation of where bottlenecks may exist. Thanks for contributing.

markbb10, I would also suggest using the NinjaScript Utilization Monitor to see if any particular NinjaScript is taking vastly more processing power than others:

• Control Center > New > NinjaScript Output
• Right-click within the NinjaScript output window > Select 'NinjaScript Utilization Monitor...
• This window will begin to populate with NinjaScript items in order of time spent processing
• Let this window populate for a few minutes and see what NinjaScript rise to the top
• Start your optimization with these top NinjaScripts

Also, I have provided a link below to our Help Guide that goes over performance tips:

• http://ninjatrader.com/support/helpG...ance_tips2.htm

[markbb10]

Thanks so much PJ for the explanation on what might be going on. It is just so frustrating. I too have a very "beefy" system that does not break a sweat with the workload but Ninja just grinds.
I will see what I can begin to compromise in my already pretty basic setup.

Thanks Patrick, I will check that out ... is there any plan in the works for Ninja development to look at their architecture and thread utilization or is this something that if you want to use Ninja this is it...

Thanks again to all

Mark

Had to edit this to include - Market has been closed now for 2 hours and 15 mins, my chart is still trying to catch up. It is working on the 15:55 candle, should be all caught up by 7pm or so.. WOW.

[habibalex]

I"m noticing the lag issue as well. I use Rithmic and while connected to their US West server I was lagging behind the market all morning. Could this be because of the tickrefresh indicator or something else? I'm trying to find a way to debug this.

[brucerobinson]

…... had lagging issues last few days with the volatility.'''.

#metoo

replaced with my own more efficient one. No problems today! Go figure...

….would you be so kind as to share? Obstinate though I am about giving Ninja any opportunity to play the 'take off all your third-party indicators card...'

I think we've all 'figured'. The end of my tether is so far behind me I'd need the Hubble telescope to see it. Plus, I have a recollection of (a?) (stock Ninja) candlestyle(s) being a culprit in the past with inefficient code (could have been NT7 though). I don't want bells and whistles (in fact, the only thing I'd like is wicks the same colour as bodies. Had it in 7 (3rd-party though)).
However I'm hoping the large hammer (see below) may crack this particular nut.

#habibalex

"

Could this be because of the tickrefresh indicator or something else?" -

for the record for this thread, I made 2 identical Workspaces, one with TickRefresh applied to the 6x 15T charts it includes, and one without TickRefresh. I can confirm in my setup TickRefresh makes no difference to the volatility-induced lagging in my setup whilst allowing increased visual clarity and flow i.e. fit-for-purpose, which 250ms is not, exacerbated in such conditions i.e. where you most want/need it

However - (and this is the bit that might be useful for avoiding problems, and/or those that understand more can use) - generally things have been pretty good (no problem today). What I'm 'fairly' sure of is that if I 'do' anything under fast conditions it is likely to induce the 'grinds to a snail's pace' and a process(es) in Task Manager showing un-responsive. By 'do something', it seems particularly such things DOM & instrument -related e.g. switch DOM tab, change ATM Strategy template, change instrument in the DOM. I know better than to do something that would require all charts to reload data for example as I know it hits cpu and memory and rendering etc. But it crossed my mind (a short trip) if an instrument is assigned to a core, is it advisable not to change that status when all hell lets loose.... Anyhow - I'm not looking to start a new tangent with that line of thought, as it is just musing and about which I know little. Simply my 'Postcards from the trenches...' feedback from practical experience is I'd suggest not doing anything like that if you can avoid it in high activity even if your cpu & mem usage is low etc etc, it will likely result in trouble is my experience, I guess for reasons of PJ's 'getting through the gate' explanation (excellent - my pay-grade level of techspeak!)

[QUOTE=pjsmith;n1056950]...

getting a newer more powerful box....

/QUOTE]
#metoo. Dual CPU on it's way. 16 cores/32 threads in an attempt to throw something at this aspect. I've only a layperson's grasp, however it seemed to me if I'm running 6 or 12 instruments and they each use a core, giving each it's own little playpen couldn't hurt. Whereas sticking them all plus everything else through four and then changing them around when Kim Jong Un's just shot his load into the skies and the VIX is grinning at 20 might be making their little eyes bulge somewhat. Soul-destroying if it doesn't result in any improvement though.

[QUOTE=pjsmith;n1056950]....

rather than new features that are almost always broken...

/QUOTE]
....don't get me started on 'Alert-gate'.
So you mean being able to set your chart background to a photo so that whilst waiting for your charts to catch up with the markets you have a picture of your dog to look at doesn't cut it with you? That comes as little surprise.

hehe, just kidding.

[pjsmith]

would you be so kind as to share?

Sure - Some of my charts don't really need the standard candle bars. I look at somthing else. Somtimes these are stock NT indicators, sometimes my own. The trick often used in this case is to make the underlying bars transparent so they are no longer shown, but the logic, at least in part, still seems to apply, i.e. every time a new bar forms, all the other visible bars are redrawn. Each time price changes, the current bar is drawn, etc. If your'e like me (and you are by the sounds of it), you have 10's of charts with low tick intervals, that's a LOT of bars being drawn. I have at least 4 on the same/each instrument like this. So, I created a new 'NullStyle' bar type. What this does, is, well, pretty much nothing! It's a shell with all the loops and drawing removed. So, on these charts, I used the 'NullStyle' candle type and voila! Noticeably snappier.

I don't want bells and whistles (in fact, the only thing I'd like is wicks the same colour as bodies. Had it in 7 (3rd-party though)).
However I'm hoping the large hammer (see below) may crack this particular nut.

Well, here you go (attached) Pretty sure you need to restart NT after installing the new candle styles before they will show. Once installed and restarted, change your data series to use 'CandleStyleOneColour'. You should now have wicks the same colour as your bars. Why NT did not enable this as an option years ago, no idea. This is a quick fix, but it is a native bar type (not an indicator). Would be nicer to have a 'set wick to same colour as bar' in the default candlestyle, which is what I did elsewhere, but this one should sort it for you. It is otherwise identical to the default candlestyle. I prefer it on large scale bars too.

However - (and this is the bit that might be useful for avoiding problems, and/or those that understand more can use) - generally things have been pretty good (no problem today). What I'm 'fairly' sure of is that if I 'do' anything under fast conditions it is likely to induce the 'grinds to a snail's pace' and a process(es) in Task Manager showing un-responsive. By 'do something', it seems particularly such things DOM & instrument -related e.g. switch DOM tab, change ATM Strategy template, change instrument in the DOM. I know better than to do something that would require all charts to reload data for example as I know it hits cpu and memory and rendering etc. But it crossed my mind (a short trip) if an instrument is assigned to a core, is it advisable not to change that status when all hell lets loose.... Anyhow - I'm not looking to start a new tangent with that line of thought, as it is just musing and about which I know little. Simply my 'Postcards from the trenches...' feedback from practical experience is I'd suggest not doing anything like that if you can avoid it in high activity even if your cpu & mem usage is low etc etc, it will likely result in trouble is my experience, I guess for reasons of PJ's 'getting through the gate' explanation (excellent - my pay-grade level of techspeak!)

Agreed - Clicking to show the list of indicators, or scrolling etc., is something to definitely be avoided in volatility, as is moving a window, etc. Nearly always causes an NT freeze up. Cannot believe NT staff do not get these issues (but are they actually actively trading several instruments in periods of volatility?)

Dual CPU on it's way. 16 cores/32 threads in an attempt to throw something at this aspect. I've only a layperson's grasp, however it seemed to me if I'm running 6 or 12 instruments and they each use a core, giving each it's own little playpen couldn't hurt. Whereas sticking them all plus everything else through four and then changing them around when Kim Jong Un's just shot his load into the skies and the VIX is grinning at 20 might be making their little eyes bulge somewhat. Soul-destroying if it doesn't result in any improvement though.

I hope this works for you and I will be interested to know the results once you get it running. NT docs do say 'takes full advantage of all cores/resources etc., but I am pretty sure when I pressed it elsewhere, I was told 'up to a maximum of 8 threads'. That's only my recollection and I cannot recall exactly where that info was (I looked for it again). I hope I am wrong, but maybe someone in NT will confirm. That's why I didn't go too mad on the processor for my new box this time, though I can upgrade it later if needed. I went in part for the highest clock speed available, which will make a bit more difference for my scenario on a single thread.

....don't get me started on 'Alert-gate'.
So you mean being able to set your chart background to a photo so that whilst waiting for your charts to catch up with the markets you have a picture of your dog to look at doesn't cut it with you? That comes as little surprise.

ROFL. I hear you on that one! GL buddy.

[brucerobinson]

#PJ many thanks
On minimising bars drawn, thanks for that expansion. I believe I'm on top of that as best I can be. In addition to the multiple charts per instrument we both employ - I have the additional burden of 3 Data Series for each chart (just so I can segregate RTH and ETH values) :-( But in the video Ninja produced on the subject they tell you to make the ones you don't need to see Line on Close and transparent for the same reasons you give.

Thanks for CandleStyle indie. Just a cosmetic thing for me and what I was used to with 7. Recently, Ninja made it possible to have the body outline the same colour, but didn't do the wick at the same time which was beyond my ken. As I'm not a coder I don't usually make comments on such, on the assumption there may likely be good reasons for what may seem to me incomprehensible, especially when it isn't important (unlike the case originating this thread).
What I do know is I had it by 3rd party indie in 7 and never caused any problems, it can be done in 8 (you've confirmed) and it is available in other platforms.

"....

but I am pretty sure when I pressed it elsewhere, I was told 'up to a maximum of 8 threads'.

" I checked the documentation before going this route, same consideration as you - do I go for fastest clock, esp. with all the Data Series, a lot of CPU processing to do in fast markets - specifically it states right up front under Performance Enhancements "

NinjaTrader 8 core and UI is now fully multi-threaded...."

and elsewhere that each instrument uses a separate core (if available, presumably). So I certainly do hope someone from Ninja stops by and confirms we have a common understanding of the word 'fully'.

Anyhow,, the proof of the pudding will be whether it is any slicker with the new kit. Here's hoping.

Cheers!

[brucerobinson]

Hello all,
I feel a sufficient period has elapsed to have some confidence in giving feedback on using Hurleydood's Tick Refresh indicator, both to all those that have followed this thread and to add to the knowledge base for anyone happening across searching the topic.

Short-form summary - solid month using in all market conditions with no issues related to doing so as far as I can tell. I'm not a coder so don't know what the symptoms of a deadlock Ninja caution against would look like but I'm guessing a freeze or something of that ilk.

I'm not posting this feedback to in any way suggest the caution Ninja draw attention to isn't merited, rather that for anyone that finds the 250ms limit a significant obstacle, I have found this solution successful without encountering problems - with my particular setup.

It may also be useful knowledge to Ninja, hopefully to allow faster rates to be opened up to the User.

Hurleydood's video demonstrations amply illustrated the problem and how TickRefresh may be used to overcome it. One of the limitations of the demo that could be directed at it and perhaps re: deadlocks is that the demo is a fairly benign setup - a chart and a DOM. What happens with a typical more resource-hungry trading workspace?

I have 6x 4k 43" monitors so that is pretty much maxed out in terms of screen real-estate for most users, I believe.
I have 12 DOMs and 36 live charts. The 250ms (4fps) restriction is a problem to me only on my fast tick charts (12 of). I set TickRefresh to 100ms on all - because I knew this refresh available in NT7 was satisfactory for my purposes.

With TickRefresh so applied, my setup has been running silky smooth and is a joy from a visual involvement perspective.

I mentioned in my last Post that I was upgrading my PC in case I was reaching my old one's limits and was contributing to performance limitations (despite no obvious indications of such, but recognising I was asking a lot with so many charts, instruments, data series (120)) and my old PC was not able to take full advantage of NT8's multi-threading (4 core 8 threads - if I understand correctly Ninja will allocate each instrument its own thread if available, hence with only 8 threads and 12 instruments this best case scenario was not possible).

New PC has 32 threads and all are being used, quite evenly. I don't know which by NT, but none ever get anywhere near 30% and in total CPU usage is always less than 20%, so there seems every indication NT8 is 'fully multithreaded' and using them all. Whatever, the system as a whole is doing so, at least. Given that other than Ninja and Windows I have almost nothing else running, seems a fair assumption Ninja is using them.

However, for completeness so that this 'wrap up' does not give the impression that TickRefresh required a higher spec PC as part of the solution, I have reason to believe it is not the case at all. The following likely also warrants a separate thread.

Additional functionality offered in NT8 allows indicators to be used on DOMs. I use OHL indicators on charts, and added them on DOMs. I have reported several times over the last 18 months various issues with the DOMs in 8, particularly when changing things like ATM strategies with the usual symptoms being slow to change from one to another, freezes, errors, exacerbated during fast market conditions etc. and slowing the DOM altogether i.e. scrolling, order entries and management often taking minutes until the DOM started to respond as normal once more, if indeed it ever did without closing down etc. The workaround was to not make changes to the DOMs in fast moving markets. However, that was far from ideal, as the time at which I want to change is usually during these conditions i.e. choosing an ATM specific to a setup unfolding.

I had nothing to suggest that this was related to the use of Dom indicators in particular. Having thrown a higher spec PC at my setup I was not experiencing any chart issues (with TickRefresh applied) but still the sluggish DOM behaviour when any changes were made.

Reflecting on what is different to my NT7 setup, I did not have indicators on the DOMs in 7. I used the Utility Monitor which showed OHL indicators as being high in the list. I have multiple in my charts so not a surprise. I removed all DOM indicators and the DOMs now respond instantly.

My layman's non-technical conclusion is it is the DOM indicators that cause the strangulation of the DOMs responsiveness, specifically related to when making changes to the DOMs such as switching ATMs etc.

It is my firm belief that throwing more horsepower at Ninja in terms of CPU speed and cores wasn't necessary for either faster chart refresh or relevant to the DOM issue. My old PC had no problem with TickRefresh at 100ms, and the DOM problem persisted with the new one, eliminated by removing indicators from the DOMs which I expect would have been the case on the slower PC also.

To my fully-acknowledged limited understanding, it smacks to me of being coding-related rather than anything a faster processor or more threads can impact significantly.

Thanks to all that have contributed.

Kind regards,

Bruce