Chart lag (AGAIN)

[pjsmith]

I hate to revisit this, but I am interested to know if anyone during Fridays spike suffered the dreaded chart lag, whilst using the new version of NT released the other day?

For the most part, I have a completely bare bones NT now. Had to strip it and minimise everything to get any reliable performance and unless we get stuff like Friday, the open, or the close, for the most part on stuff like NQ (volatile), it's acceptable (not perfect, but acceptable).

I've been running the chartLag script for a while, nmonitoring latency. During that down move on NQ, it was topping out around 80 seconds lag. I've attached a screen shot showing the lag in excess of 60 seconds. It stayed there for a few minutes, at least, then resumed it's normal 120-200 ms (I'm transatlantic for data).. Of course, the _perfect_ setup occurred, but I'm looking at data over 60 seconds old, so no point even looking at the mouse! There's nothing left I can optimise. I've even removed / enhanced NT bar types to make them faster. The biggest resource use according to ninjascript utilisation manager are things like 'candle type'.

Now, I'm not looking for response from NT - I know they cannot fix this. But, there was comment that the new version just released may have some performance enhancements to resolve this. So, just looking for any feedback from anyone who is using the latest, if they had any issues with lag Friday.

For my part, I tried the new version, but very, very, quickly, found many issues with charts freezing, loads of errors in the log re NT drawing objects, etc. So, I rolled back to the old version and all OK. I tried the update twice with same result. Would not run more than a few minutes during live session before charts froze. 8.0.17.2 64-bit I am on now, seems reasonably stable and certainly does not have the issues the latest one does.

Just clinging to hope the new ver might help with performance, and once the gremlins are sorted with that, I might have a path forward with NT.

Thanks.

[NinjaTrader_ChelseaB]

Hello pjsmith,

Can this be reproduced with 8.0.19.0?

If so, can be reproduced with all workspaces closed in a new blank workspace with a new chart that has no indicators applied?

[Lancer]

pjsmith, Using the ChartLagTime script to monitor chart lag, there is still significant lag during fast market times with v8.0.19.0. The lag seems to clear faster than with v8.0.18.1, but I don't see much of a difference beyond that. On Friday, US equities charts lagged at the open as usual, and later in the day the ES chart lagged by up to 10 minutes after one of the news events. ES is always the worst though by far, then NQ. At the same time that ES and NQ are lagging by a minute or more, other lower volume or lower trade-rate markets have no chart lag at all. The subject workspace has 6 markets charted, multiple timeframes tabbed for each, with indicators on all, and no other open workspaces.

PC resources during these fast-market events, showed plenty of unused CPU, GPU, and memory resources, with individual CPU threads around 40% utilization average (high compared to normal market). Overall, CPU (3.5 GHz 6-core Xeon) utilization by NT ranges between 5% and 35%, depending on market activity. Not sure where the bottleneck is, but my PC certainly has more resources available for NT to use. I'd increase PC resources to whatever hardware makes a difference, but NT is not using anywhere near all available resources as-is, and other users report the same with higher-core CPUs than mine and more memory, so there doesn't seem to be any point in throwing money at hardware only in hope that it helps.

Otherwise, in all aspects, v8.0.19.0 has been stable, with no issues, no freezing, and no log errors, with the same results on a second PC with similar hardware and NT configuration.

ChelseaB, my guess is that there is no chart lag using a blank workspace with only one chart and no indicators, but that configuration is not a realistic use model.

[habibalex]

Lag is definitely showing up for me also. We need a way to stress test our indicators with the ability to test OnMarketDepth as well as OnMarketData through the external connection. I made a program that used on the external connection to feed data to OnMarketData using last, bid, ask updates and I am unable to reproduce the lag after playing back data that caused lag while live. I'm pretty sure the OnMarketDepth level2 data is the source of lag in my indicators but I have no real way to test it.

Also, as far as stability goes. My charts keep freezing on v8.0.19.0. I am using an indicator that has a parallel for loop and that is probably the one causing the freeze. Downgrading to the previous version v8.0.18.1 resolves the issue so I am sticking with that for now. The error was in OnRender and "Attempted to read or write protected memory".

[pjsmith]

Thanks for the responses. So it seems new ver is not significantly different, I guess.

ChelseaB, thanks for the response, but I'm not even going to try that exercise (which I have many times before). As stated, I cannot run .19 and continue to work. .19 charts freeze within a few minutes use for me, so I am unable to trade or avoid even more risk than usual, if I install it. In terms of replicating the lag with an empty workspace, 1 chat, etc. I suspect it works, but that setup is no use to me. (or most others, I suspect). It is NOT how most people use platform and frankly, I don't see it proves anything at all, other than NT cannot cope with mildly complex (if you can call 10 charts complex) setups during volatility. If I cannot run my monitors with my 4 charts per instrument, then the platform is useless to me. I currently have 16 charts open.. Actually very little in way of 'indicators'. Ninja utilization monitor shows nothing significant and top resource users are NT supplied code, like the candle type. Given that, not sure the finger can be pointed at any 'custom' changes my side.

[NinjaTrader_ChelseaB]

Hello pjsmith,

Our process is to work through steps to find where the behavior may be coming from.
Attempting to analyze everything in your system without testing is basically impossible.

We start by testing NinjaTrader by itself to see if there is an issue with the NinjaTrader platform we need to report to our development.

If the behavior cannot be reproduced on a single chart or with several charts that are not using any 3rd party indicators, then we start testing the 3rd party indicators.

Sometimes we find specific bar types and indicators with poor design choices (such as creating large numbers of objects in the wrong methods) that can lead to undesired behavior.

From our help guide:
"NinjaTrader 8 provides a development environment allowing low-level access to 3rd party developers to build integrated indicators, drawing tools, automated strategies and more. An Add-On with software bugs can have adverse effects on the entire NinjaTrader application. These add-ons also natively run on your computer, therefore, its important to only install 3rd Party Add-Ons from sources you trust.

The following symptoms post installation could indicate an Add-On is installed causing negative impact:

• Windows become slow or unresponsive to user interaction
• Market data becomes unusually slow to load or update
• Standard features fail to work as designed
• Other scripts fail to work as designed (custom scripts that work on their own may conflict with each other)
• Lost connections from market data providers
• Error messages are generated at various times
• The entire application shuts down abruptly and without warning

If you run into any of the above symptoms post installation of a 3rd Party Add-Ons please try uninstalling the 3rd Party Add-Ons to see if the problem goes away and contact the 3rd party developer for support."
https://ninjatrader.com/support/help...tallingAdd-Ons

There is a major change in 8.0.19.0 that enforces better coding design that may cause some indicators that have made poor design choices to no longer work and will produce errors in the Log (and trace).

From the help guide:
"Attention developers:
•When working with D2DFactory to create Direct X resources, this must be done from the charts UI thread otherwise there will be a performance impact. This is now enforced in 8.0.19.0. If your script does not follow this policy it will not be compatible with 8.0.19.0 and a log error will be thrown. Please contact platformsupport [at] ninjatrader [dot] com if any questions."
https://ninjatrader.com/support/help...8/8_0_19_0.htm

Please let me know if you would like investigate the behavior you are reporting so this may be either discovered to be an issue with the computer hardware and size of workspace, or an issue that needs to be reported to a vendor, or if its a NinjaTrader issue that needs to be reported to our development.

[brucerobinson]

#pjsmith
"

I am interested to know if anyone during Fridays spike suffered the dreaded chart lag, whilst using the new version of NT released the other day?"

I'm guessing you, #ChelseaB et al. here will be interested to know (thus giving you a Ninja hOpium fix) if during Friday's spike, using R.19 and similar real-world workspaces as described herein and no 3rd party indicators didn't suffer the dreaded chart lag (& likewise across the pond).

Me.

I'll elaborate & offer some further info/comments/detail FWIW a little later.

If anyone has technical understanding of the D2DFactory reference and could comment (in layman's language - I investigated MSDN etc but above my paygrade), it may help with transparency. Perhaps thinking along the lines of 'what if changes were made in 18.1 which uncovered poor design choices in the stock platform (no 3rd party indicators), throwing Log errors with gay abandon. Hence a major change in 19.0 that enforces better coding design now adopted in the stock platform, the poor design choices in the stock platform having been rectified, but will likely cause problems with others'. Just a hunch (with a lot of supporting information & rationale).

Best,

[NinjaTrader_ChelseaB]

Hello brucerobinson,

The error means that objects that attached to a render target are being created outside of OnRenderTargetChanged().
All objects involved with a render target or the (NinjaTrader.Core.Globals.)D2DFactory (for example when creating PathGeometry) should be initialized in OnRenderTargetChanged() or (OnRender()).

Below is a link to an example.
https://ninjatrader.com/support/foru...606#post789606

[brucerobinson]

Hello ChelseaB and thanks,

I believe you're referring to the errors specified by habibalex. I'm not a coder (an increasingly rare species around here) nor looking to write or debug anything (but thanks for the link - I learn something new every day, whether I want to or not),

I was referring to "

When working with D2DFactory to create Direct X resources, this must be done from the charts UI thread otherwise there will be a performance impact" - I've no doubt it is babyspeak to a developer, but it is (almost) impenetrable to me.

Simply put, to me you seem to be saying, correct me if I'm wrong, previous to R19 it was possible not to blahD2DFactoryblahDirectXblah.'..from the chart's UI interface', but not best practice to so so and could result in poor performance.
.
Changes in R19 mean if your were/are doing that, now doing so simply won't work rather than a performance hit and will produce errors. So take heed, you'd better stop your wayward practices and blahD2DFactoryblahDirectXblah.'..from the chart's UI interface from now on.

Correct?

Kind regards,

[NinjaTrader_ChelseaB]

Hello brucerobinson,

While I cannot comment about what has changed under the hood of NinjaTrader, I can say that in 8.0.18.1 and previous it was possible to initialize some objects in a thread that is not the ChartUI thread where rendering is done.
NinjaTrader has made changes to how rendering threads work and because of this our development is now enforcing that all objects related to the RenderTarget be created in OnRenderTargetChanged() (or OnRender()).

The short of it is. Initialize objects in OnRenderTargetChanged(). This would have avoided the performance hit to begin with and will avoid the D2D error.

[brucerobinson]

Hello ChelseaB and thanks, I believe that may be helpful.

I'll follow up with more later

Kind regards,

[brucerobinson]

Good news

I have been using R19 pre-release build because 18.1 trashed my stable 17 setup

My Workspace: stock platform plus a couple of Ninja add-ons. RTH & ETH Sessions r

ange highs and lows

for stock indices, CL, GC, NG and some currency Futures.

Daily, 15m, 5m, 1m etc.

Bare candlestick charts and Volume indicator, Current & Prior Day OHL indicator. That's it. Trading 101 setup,

I've attached a grab.

. Not an 'overly complex' workspace imho. The only non-Stock Platform add-ons are the range shading and Rollover Notification - both Ninja add-ons.

I use #Hurleydood's TickRefresh I removed it and both the non Stock Platform indicators - Barebones stock platform and R18.1 still crashed, lagged etc. where R17 didn't.

I can now report R19 pre-release worked for my setup but with some lag in fast markets, and now R19.0 final build without lag. In the first instance stock platform only, now having added back the two Ninja add-ons and TickRefresh. No errors, no lag, no crash with all functionality in all conditions. A few more days testing yet, but I'm close to saying even extreme volatility has no discernible effect on my setup whatsoever.

My workspace is now only 8 instruments as a result of trying to get R18.1 to work (i.e. I cut it back, but it didn't stop crashing), whereas stable R17 setup had 15 instruments.

Crosshairs remains an 'issue' - in my 8 instrument setup it is not, but in my 15 instrument setup there is perceptible lag in high volatility (#Hurleydood made a nice video demonstrating this). However in multi-monitor workspaces it always has been problematic since NT7. Any movement of Global Crosshairs takes GPU to 100%. This may be by design to maximise use of available resources, I dunno. I can work around it by using Global Crosshairs sparingly.

Quote #ChelseaB "

While I cannot comment about what has changed under the hood of NinjaTrader, I can say that in 8.0.18.1 and previous it was possible to initialize some objects in a thread that is not the ChartUI thread where rendering is done.(and from the Release Notes advisory a consequence of this could be poor performance and errors, crashing)

NinjaTrader has made changes to how rendering threads work and because of this is now enforcing that all objects related to the RenderTarget be created in OnRenderTargetChanged().

The short of it is. Initialize objects in OnRenderTargetChanged(). This would have avoided the performance hit to begin with and will avoid the D2D error." Unquote (itallics text added)

It appears to me that that the stock platform must have been guilty of this in R18 because it produced everything referred to - D2DF, DirectX, citing Paint, Brush, Draw, Object, OnRender etc etc etc & crashes galore.in my Setup - no 3rd party anything involved. Now fixed in R19 and with enforcement.

Releasing 18.1 (I even skip .0 releases so others can uncover the problems) with such issues cost me around 100 hrs and a lot of inconvenience with crashes, reporting them, taking off Ninja's own add-ons etc.. A disgrace.

I watched the presentation over on Futures.io by Brett Barratt on R19. Most was introducing new tools but a couple of things to give hope to those here - he stated the aforementioned changes in R19 had resulted in 20-30% improvements in performance reported by his team, depending on circumstances. I have no reason to doubt his word so that is encouraging to hear.

'Depending on circumstances', there has been questioning of what is fit for purpose for 'real world' setups (in this thread for e.g.) In the Q&A session, un-prompted Brett specifically raised and asked for feedback on multi-monitor/multi-instrument use of NT8 looking forward. Maybe the squeaking gates are being heard

(A minor correction - I do have a recurrent error in the log which is new, I believe, to R19 which I wonder if related to this topic OnRender InitializeIn etc. Indicator OnBarUpdate on bar 23674 (or whatever) calling thread can't access 'cos different thread owns it. I typically get a block of msgs all within a few seconds. They seem benign so not much bothered, but haven't seen before & nothing new (except R19). Just for completeness. Not wishing to hijack #OP's thread for this - I'll put in a ticket if I think it warrants it, but no impact I can see)

Update: same to report today, in extreme volatility things that would always have broken Ninjatrader in the past during high volatility (NT7 or 8) - move a fast chart around to cause re-rendering, change an interval, change days loaded, scroll, re-size axis etc all absolutely verboten with my setups, would have caused to either take forever, timeout, freeze or crash. Deliberately did all this on indices during Open and oil news & can't break it or make it lag.

Hope this helps.

Kind regards,

[pjsmith]

Interesting stuff, Bruce. Thanks for the detailed post. I can't run .19. It's unstable for me, but I have made a breakthrough in lag, in .18 over the last week - And it's not my code that has been at fault. Not trying to be a tease, but I'm not going to post any details until I have carefully tested it, and .19 might even make it irrelevant, but I have found (I think) what causes lag in my setup, and a workaround, which requires changes to a lot of NT's code. I'm been testing it in crude form yesterday, and slightly more refined today, and whilst the periods of volatility were there, and the lag observable, my platform adapted. where's before, it would have likely been stuck in that state until it caught up, which could be some time. I will test .19.x when the next version comes out, and my version until then.

Like you, I've spent 100's of hours on problems and workarounds for this platform. That's a lot of cost...

[Alfred]

Here is a screen shot from a 10/01 spike....no lag showing.
2 workspaces ...seconds charts and 3 indicators.....simple stuff....
My machine has an i3 and 12 MB of ram.

[pjsmith]

Alfred I think simpler works better for NT8. It's when you start messing with tick data and tick charts it all falls apart in volatility. If your method does not need that, advantage you