Fast market and slow NT

[percma]

Hi
I'm wondering I there is something that I could do to solve an issue. I've recently that when we have a fast market NT will lack behind the actual price. I'm running on the same machine NT and Sierra Chart and at one point price where 3-6seconds behind.
Let me say that both were using the same data feed.
Is this due to a settings issue in your opinion?
Any suggestions?
Thanks

[Lancer]

Some other threads on this issue:
Chart Lag (Again)
NT8 Freezing During High Volume Periods
NT8 Lagging
Charts/Price Lagging with CQG/Continuum
NinjaTrader 8 Lagging
Lagging NinjaTrader Chart by 8-10 Minutes Following FOMC
Any Plan to Utilize Multiple CPU Cores?
Charts take over an hour to load
Questions about NT8 chart performance tips
Performance Issue - Code Comment, Please
Laggy Chart Due to Number of Drawings
Indicator Lag during Period of High Market Activity
ChartLagTime - What kind of chart lag is acceptable?
All my charts are lagging!
Super DOM freezing in fast markets
News events?
Delayed feed when compared to another platform using the same data source
Nt8 lag lag lag
Chart Lag
New Computer Spec Requirements
NT8 - the great and the horrible
Ninjatrader slow and laggy
Lagging screen
Improving performance by reducing CPU load
Chart is lagging big since the new update
How to create a lag free workspace
Lag problem
Chart lag is inconsistent
NT8 Charts Lags and Delays - Need a solution ASAP
Chart update frequency
NT8 - Jump in Memory Usage
Lagging Issues

Indicator script to monitor chart lag (import to NT8 and add to chart):
ChartLagTimeV3.zip

The script originated from user WhiteRhino as ChartTime here, and I did a slight modification, renaming it ChartLagTimeV3.
Zac White of Shark Indicators further modified it and renamed it ChartLagMeasurementV1 here.

[NinjaTrader_Tyler]

Hello percma,

Thanks for posting.

This is generally a symptom of poor PC performance due to unoptimized or overly complex workspaces.

I have provided a link below to our Help Guide that goes over performance tips. I highly recommend reading through this list and making as many optimizations to your workspace as possible. If you have multiple workspaces running simultaneously, you'll want to close any that are running in the background.

• http://ninjatrader.com/support/helpG...ance_tips2.htm

If you're still seeing delayed charts after taking these steps, please send me your log and trace files so that I may look into what occurred.

You can do this by going to the Control Center-> Help-> Email Support

Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

Please reference the following ticket number in the body of the email: 2275604 ATTN Tyler

I look forward to being of further assistance.

[James108]

Hi Percam

Some early testers of the upcoming version 19 have reported that Version 19 is much better in fast markets. Keep an eye out for the official release and try it and see if it solves your issue.

[brucerobinson]

Hello Percma,

"This is generally a symptom of poor PC performance due to unoptimized or overly complex workspaces."
Follow Lancer's links. You may also visit Futures.io for similar (search user Hen322, simple workspace, lagging & crashing. Put on 19 internal build. I was told no-one else was experiencing crashing with 18.1).

This sort of disingenuous Ninjadoublespeak denial and deflection incenses me after many several hundred hours spent so can't help myself.
"No it isn't, it is a symptom of the coding of the platform". From what you describe, you do not say your PC is performing poorly. Typical Ninjaesque response which originates at the top, a cultural thing imho (search my name on Futures.io where the NT8 OHL indicator caused lagging (some 2 years ago), An update fixed that, but there were still other Ninja problems - Ninja's fearless leader quoted my post re: the lagging fixed 'glad we could solve your problem' - but ignored the rest. No, you fixed YOUR problem.
There is nothing overly complex about my workspace. I want to see the commonly traded Futures, 5min, 1min and a Tick chart with ETH & RTH ranges. That's it. I use the stock platform Current and Prior OHL indicators and, since their introduction, TrendLines. That's all. I upgraded my PC from a mid-spec to a business grade behemoth twin Xeon 32 thread workstation. '...symptom of poor PC performance' indeed. It's not a symptom of poor PC performance, its poor software performance. 'Generally...' is what I call a weasel-word, covers and deflects from, rather than addresses, a multitude of sins.

"Some early testers of the upcoming version 19 have reported that Version 19 is much better in fast markets".
I am one of those early testers of R19. My setup was working tolerably with 17.2. R18.1 destroyed it - lagging and crashes daily. Stock platform. I can confirm that I do not have the crashes Ninja (read carefully, NINJA, not third-party indicators nor anyone else responsible) introduced with R18 and that performance is satisfactory in fast markets. HOWEVER, note that in order to achieve this I have had to reduce my workspace from 12 instruments to 5. With 12 instruments R19 lags - charts/DOM, and Crosshairs. Worse than 17.2. I don't know whether it is better than 17.2 would have been with 5 instruments. But with 5 it is satisfactory, with 12 it is not.

My workspaces are always optimised with performance considerations paramount to the point of obsession, and not 'overly' complex. They are as simple as they can possibly be to see the ranges of the overnight and day sessions of the most commonly traded Futures.

The bottom line is that what you seem to be reporting is that with Ninja and Sierra running in parallel, and the same datafeed, Ninja lags in fast markets where Sierra doesn't. Unless you have two very different setups making such a comparison ill-conceived (and why would you, you would recognise that and take steps to address it yourself first I think, I've seen your posts previously), then we must conclude your setup is overly (by whose defnition?) complex for Ninja but not overly complex for Sierra, and requires a degree of optimsation on Ninja (if indeed this can achieve a fit-for -purpose outcome) which has not been necessary on Sierra. 'generally.....'.

Call it out when you see it :-)

Support will undoubtedly assist you best they can, as Tyler has offered. But let's tell it like it is.......
(Probably won't assist me any further ;-) but 100s of hours takes its toll :-[ I am still getting errors with R19 that I didn't get with 17.2, citing 'Indicator'. Stock platform remember, only Ninja indicators. Support can't help). Sigh. Maybe R20 is the grail we're awaiting.

Good luck

Kind regards,

[NinjaTrader_Tyler]

Hello brucerobinson,

Thanks for your note.

When a symptom of poor performance is reported, it is Platform Support's responsibility to do what we can to assist the client with getting the best performance possible. Often that involves making changes to workspaces. Certainly there are other factors involved, and my initial reply to percma was intended as a means to get the troubleshooting process started, rather than a definitive diagnosis. We're always available to assist if you'd like us to take another look at your setup.

Don't hesitate to reach out with any additional NinjaTrader items we may assist with.

[Lancer]

Chart lag is the biggest issue with NT, and I trust that NT developers have that fix prioritized accordingly. When charts lag, NT is unusable, and then all other details don't matter. Using 8.0.20.0, there was lots of chart lag today after the China tweet, while NT was using only 35% CPU max., and only 4% GPU max., with lots of memory available. That is with the NT process priority set to "High Priority", up from "Normal". Wouldn't be a problem if NT lagged only if 100% CPU, because users can fix that with hardware upgrades, but as it is now, users with even 16 core (32 thread) CPUs report chart lag with CPU utilization less than 15%. Some chart lag events recorded today, ES -1,500 seconds, NQ -500 sec., CL -400 sec., GC -350 sec.

[brucerobinson]

#Lancer - FYI: my 'overly complex workspace' (yeah, right Ninja - 'twas the Platform, stupid - my setup is LESS complex than it was in NT7, far less indicators and stock platform only), with #Hurleydood's TickRefresh indicator to make charts fit for purpose in fast markets (which makes no difference to platform performance for me - I simply have a 'no Add-ons' testing setup should I need to prove it & deflect the 'it's your 3rd-party add-ons' mantra), was completely un-fazed by The Donald's announcements' effects on the markets. Seriously, not a stutter. The usuals - Indices, GC, CL, NG etc. all going ballistic on charts and Static DOMs, various mixes of OnPriceChange, OnEachTick updating etc. 40 charts with multiple Data Series & 12 DOMs, 3x that running in the background, double that with Currencies in another open Workspace and not a glitch & smooth as butter (all aspects of the platform, unlike the markets).

I do have a behemoth of a workstation but I don't believe it's necessary, I believe my old one would cope - it did with more demand from my NT7 setup. Also, my internet connection is 'narrowband' (2mb) - latency is ok however, although I am the other side of the pond so I'm hardly 'co-located'.

The milestone event for my setup was .18 when Ninja re-wrote the 'D2DFactory off single UI' stuff (all Klingon to me but I can search MSDN Forums etc to get the gist), Release Notes informing 3rd-party add-on Developers that some naughty incompetents had not been following best practice which I have no reason to doubt, and thus the cause of Users' performance woes, aspersions being cast and bringing the Platform into disrepute. How very dare they. Thus NT had tightened the ship in order that these ne'er do wells clean up their act, woe betide those that didn't, they were being run out of town for sullying 8's name...
Problem with that story, is that when Ninja rolled out .18.0 it totally cratered my (totally stock platform) setup vomiting errors like a teenager after their first pub crawl - yes, you guessed it, D2DFactory errors daily by the bucketload. So it kind of suggests it wasn't just those 3rd-party developers that weren't following best practice on D2DFactory or such like causing poor platform performance - doesn't it now..... Hoist by one's own petard. Transparency? Err, deflection, opaqueness and obfuscation, me thinks.

Since those Platform D2DFactory changes were made in 18.0 and the issues they caused in the stock platform fixed with Release of 18.1 and .19, the difference for my setup in platform loading times, operations, speed and any form of lag (chart, DOM etc) has made it almost un-recognisable. Blistering and un-breakable. There may have been other changes made also, I am not to know, but that was the biggest platform step-change in performance for me to date by a country mile.

I don't have any tick replay, Level 2, order flow or anything of that nature,

I'm just a simpleton

(I mention because these seem to get attention re hit to performance). But can report 8 is now doing what it should, how it should (should have from the outset), for my setup. Doesn't solve your problem, but hope gives you some cause for hope and fortitude to pursue.

I haven't upped to .20 yet, as I'll await .20.1 and meantime watch the Posts on 20.

I'm re-assured to see an issue with Global Linking across workspaces fixed in .20's Release Notes. I won't get back the 40+hrs I lost diagnosing shoddy and poorly thought through development for them unfortunately. Nor the 20+ spent instating it, un-instating it, and, in due course re-instating it!

Best of luck

[Lancer]

Bruce, impressive workstation there (dual Xeon 32-thread CPUs), and very happy to hear you have found a combination of hardware and D2DFactory code fixes that resolved your performance issues. I assume you are running the ChartLagTime script to measure actual realtime chart lag status. I'm running a single Xeon 6-core (12 thread), with dual GPU graphics and 6 monitors. NT uses only 35% CPU max during fast market times when charts lag, with plenty of graphics and memory resources also left unused, so I suspect that upgrading to an extreme CPU system would just be a waste of money. For example, in another case of extreme hardware (single 16-core, 32-thread), that user reported less than half the max. CPU % utilization for NT compared to mine, with lots of other unused resources, but still had chart lag.

As to the workspaces here, only one is open at any time, and typically have six charted markets (10 tabs per market), two DOMs, with five additional markets uncharted and listed in the Market Analyzer only, and seven market internals symbols also listed in the Market Analyzer only. Workspace scripts cover a range of essentials, with some multi-timeframe and some OnEachTick, but are not complex or lengthy, with no D2DFactory elements. There are no errors in log and trace files. Internet bandwidth or latency is not an issue. The only problem is NT chart lag during fast markets.

I haven't experienced any issues introduced by v8.0.20.0, so maybe give it a go. Best of luck to you as well.

[brucerobinson]

For Post subscriber guys and gals that want to cut to the chase - see 'What I did,' below. I believe ti should be of interest and help to all to know, even if it doesn't resolve your issues, including future stumbled upons.....

#Lancer - reason/rationale that led to PC upgrade was I'd 'upgraded' to 8 partly because where I'd got to on 7, not multithreaded, was limiting, old PC was 4 core so thought 8 would unleash more, but then found 8 uses 1 thread per Instrument, I have 12 running in the active workspace and 12 (some duplication) in the open but inactive workspace. I didn't get into the micro of whether the same instrument in two open workspaces uses 2 or 1 threads, I just smothered it in the most threads the PC industry had to offer and said if it can't work on this it's not my PC which is the problem, you could design, model and render the International Space Station on it, or whatever, 7 could do what I was asking on one core.. 8's multithreading and Windows seem to do an admirable job of playing nicely together and like you I never see hardware resources troubled, rarely disturbed from their slumbering 5-10%, perhaps 20 with a Donald.tweet

ChartLagTime script - no, wasn't using nor previously. I'd kinda worked on the basis that with #Hurleydood's TickRefresh allowing me to have fit for my purpose chart refresh, and given that I use DOM's next to them and don't have a problem with DOM refresh or lag, if I couldn't see or perceive a difference between the two I shouldn't be concerned, that was an acceptable yardstick.

However, having got to this happy state, and others have not - I thought I'd give a few more hours input (in percentage terms it's a fraction of what it's taken to get here) for what I hope to be my swansong (even if it may in fact also be my soliloquy)

What I did:.
(for quick summary, my setup has a gazillion instruments, charts, screens, data series, DOMs etc. The objective is very simple and straightforward - day trade the common Futures' Session ranges, primarily Currencies and DAX during London; US Indicies during NY. That's it. Achieving that with 8 stock platform requires what Ninja refer to as an 'overly complex workspace'. It's a simple ask.....).

I put ChartLagTime on all my fast Tick charts (these are my fastest charts - e.g. 15T for NQ) with your Default Settings
Most of the time, most of them remained Green most of the time, ranging maybe 250-350ms (goes Orange >300) lag

I did a bit of experimenting to 'stress test' and 'base/best case' by which I mean two 'full' workspaces open i.e. one full one open in the background rather than only one at a time - in that case 24 instruments 'running' but half are duplication in the background, 12 live. 28 live charts in the live charts i.e. being rendered, 3 times that in Tabs. 9 Data Series per set of visible charts per instrument. So lets say maybe close to 100 Data Series to crunch for the 'live' displaying charts, and two times that in the Tabbed charts. Only 1 per Instrument per live chart is drawing as the others are set to LineonClose and Transparent so not rendering candles. The point is not 'Workplace complexity Top Trumps' because I would far rather have one chart and data series per instrument per timeframe but the stock platform cannot do that.

I then set up a single 'my setup' (i.e. multiple data series, OHL etc 15T candlestick chart in my Untitled Workspace, with a Priceline and ChartLagTime.and got around 250ms. So negligible difference between a 12 instrument full bifta workspace and a single chart. I then stripped off everything one at a time down to one Data Series/one candlestick chart, nothing else - bare-*s* naked (Ninja's 'real world' test setup :-). Still around 250ms.

There is barely any difference in chart lag time reported between minimum load case where I have only the Untitled workspace open with 1 Data Series and 15T Candlestick chart, compared with maximum load 2x 12 Instrument, gazillion Data Series and charts, each with 6xOHL indicators and the Repeater 'shading' the Sessions with Dynamic Range.

I then sat through today's Open. Not as volatile as some, but still pretty tasty nonetheless. Same. I couldn't say I could perceive a difference in lag around the increased volatility of the opening period. ChartLagTime showed me there was, because given that they were operating around the 300ms mark and this is the setting for colour change, they were orange slightly more than they were before the high volatility, sometimes, on some. But still Green in the main. If there had not been the colour change tell-tale I wouldn't have been able to call it as so.

There is barely any increase in lag time due to increase in volatility. The minimal increase in lag time that there is during high volatility/activity, is little different whether one Workspace open with one chart/data series/no indicators, or 3 Workspaces open, 24 instruments a gazillion.... - you get my drift.

The only time I saw lag over 400ms (just displaying 'Red' momentarily on a few instruments at >1000ms) was when switching Workspace i.e. inactive one catching up loading data, which one would expect, the waking up workspace is catching up with data (or atleast as far as I understand it)

Not quite sure why I'm seeing a best of around 250ms with only 1 chart, when I can ping CQG's servers at 15ms. But I'm only looking for fit-for-purpose. My charts refresh at 250ms thanks to #Hurleydood and my lag it seems is of a similar order, made visible thanks to your little goodie (and understandable to a non-coder such as I, thanks to your informative coder's notes therein!) .

Good enough for me, in the context of trade execution times, my brain processing times (ever increasing) etc. I'm not looking for HFT, scalping etc

For my use/setup, it seems it is at last working as it should.

Hope this info may in some way help get to the same position with yours and others, or in some way be useful to you..

No apologies for the long post - if I'd had time I'd have written a shorter one ;-)

I believe my work here is done.

Kind regards.

[Lancer]

Bruce, Same here, yesterday was a very low lag day, with lag occurring only for a bit at the open and at the close, less than 20 seconds lag when it occurred, so not much opportunity to observe lag measurements.

Regarding operation of the ChartLagTime script, the script measures lag per instrument, not per chart. Regardless of the chart it is added to, the lag measurements will be the same for all charts of the same instrument in the workspace, so you can put it on your tick chart or your minute chart, or both, and get the same result if both charts are the same instrument. You have multiple instruments charted, so with ChartLagTime on a chart of each instrument, you will see different lag results for each instrument. For example, symbol ES has the highest volume and trade rate, so that is always the highest lag in my workspace, then NQ. Also, for accurate readings, it's essential to first synchronize the computer clock with a time server, and repeat synch on a regular x-hour schedule.

[brucerobinson]

Lancer, thanks for helpful clarifications/further info, hadn't grasped it was chart independent (name threw me - should be InstrumentLagTime :-),- but had grasped it was instrument specific as like you I see (slightly) differing lag values. I'd read up before installing so had synched my clock (have a little widget that does it periodically anyhow).

I left ChartLag om for the time being.
Today, it is reporting around 750ms all round. - 2-3x yesterday's
Haven't changed anything.
Doesn't matter whether it is one workspace/instrument/chart/no indicators, or one full-fat workspace, or two full-fat workspaces
Ping to CQG is still same as yesterday at 15ms

It's not a problem, but surprises me as can see no reason

Kind regards,

[brucerobinson]

update: just had 3 'slow downs' in close succession (around 11:45 EST where the charts slooowed to a standstill as ChartLagTime rose (maximum out of the three instances was around 30 secs) then caught up again to a stable 750ms -ish on all charts.
A couple of minutes later, same thing. Then once more.
This did not correlate with anything of note transpiring in the markets, as far as I could see
In between, I did a quick ping whilst this was happening, and circa 600ms whereas yesterday and earlier today 12-15ms

Any ideas? Does it suggest my internet connection having a couple of flakes, which then has to be caught up with - data packets transmitted, loaded, charts drawn etc and then business as usual (I didn't notice/think to look at pc resources but everything remained responsive in the platform and otherwise in the PC?

Kind regards,

[brucerobinson]

update - a bit more info for the knowledge base

The apparent increase in 'lag' time was due to PC clock drift (even in just a few hours since last sync). Synching brought all back to ~ -100ms

This has been to all intents and purposes unaffected by number of workspaces, charts, data series or market activity for my setup - the difference between light case single chart single instrument/data series in a slow market, and fully loaded multiple open fully populated workspaces during frenetic market activity seems for me to be of the order of tens of milliseconds..

The only negative event was the 'slow downs' reported above - but I'm tending toward this being my internet connection, it can be flaky at times, and a direct Ping though Powershell at the time (i.e. from the OS) was also slow.

I'll keep ChartLagTime on and synch'd for a while longer so I can 'see' it.

Thanks Lancer.

Kind regards,

[Lancer]

Yes, if the PC clock is not synchronized with a time server, it is noticeable by that uniform increase in variance that you saw over all instruments. If not monitoring chart lag with a script, there is only perception to suspect a chart is lagging, then a visual comparison between chart price and DOM price to verify, so even if all lag issues appeared fixed, I would still run the monitoring script to be sure of an alert during lag events, from whatever cause. Normal market here on the US west coast is less than 100ms variance indicated as well.

Are you running any multi-timeframe scripts, or multi-indicator? In the Utilization Monitor, those are listed at the top in total time (ms).