Chart Lag

[brucerobinson]

Hello,
All my charts are lagging today by several minutes. Any ideas?
Short story -
My workspaces have ChartLagTime indicators on fast tick charts.
Everything has been working normally, no lag.
No changes,
Today, all charts lagging by minutes,
Closed all workspaces & rebooted Ninja.
Opened New Untitled (empty) Workspace (only).
Loaded a new tick chart RTY - no indicators or anything, no template
Charts are 5 minutes behind - I don't need ChartTimeLag - I just look at the time of the last bar on the chart and compare it with my watch.

In my workspaces - all charts, all instruments same magnitude of delay

In a bare-naked workspace with 1 bare-naked chart, same thing.

My internet connection is same as normal ping etc

Kind regards,

[brucerobinson]

So I rebooted my PC (it starts from a clean boot every morning anyhow).
Single workspace/naked chart stated off in sync.
Loaded trading workspace - cumulative lag build up.
Closed trading workspace
Cumulative lag build up on naked workspace/single chart

This has been reported before (and for which reason Hurleydood wrote ChartLagTime indicator.

I haven't seen it in months through turbulent markets

To re-iterate - 'bare naked everything'
1 new workspace, 1 new chart, 1 data series candlestick OHLC chart no fancy chart styles nothing, no indicators, no 'anything'. stock platform only.
3 minutes lag after up for 15mins.
Ninja servers? Broker servers? Any ideas?

Kind regards,

[NinjaTrader_PatrickG]

If you've truly conducted a safe mode or 'blank' workspace test, then this indicates two possible root causes:

• Your data provider is supplying delayed data
  • I observe no delay on my end when comparing a CQG-based feed to my PC clock
  • https://www.screencast.com/t/Ywo88PCazD
• The connection between your and your data provider is experiencing issues
  • Syncing your PC clock, power cycling your router/modem and other basic networking steps can sometimes mitigate or eliminate this symptom
• Some other variable unique to your PC and/or the connection between your PC and your data provider

So I may accurately assist you, please answer all of the following questions:

• What version of NinjaTrader 8 are you using? Please provide the entire version number. This can be found under Help -> About (Example: 8.0.?.?)
• Who are you connected to? This is displayed in green on lower left corner of the Control Center window.
• Do you receive an error on screen? Are there errors on the Log tab of the Control Center? If so, what do these errors report?

[brucerobinson]

Hello Patrick and thanks.
v21.1
AMP
No errors in logs (relating to this - there are errors that appear daily but have for ages and without this issue and don’t affect performance).
I haven’t done a ‘Safe Mode’ (didn’t think to) but have done as I described - single chart etc etc. But I will for avoidance of doubt.

Then on to PC clock sync - I have Windows plus 2 ’widgets’ in addition
Then router reboot

No ‘obvious outward signs’ from any of those, & been solid for months.

Thanks for the step-by-step, will follow & report back....

Kind regards

[brucerobinson]

Hello Patrick,
I am just back at my PC - left running with the single workspace chart.
This had returned to sync.
I re-loaded my trading workspace (which includes one ChartTimeLag on each instrument).
After the loading 'delay', and into 'steady state', no lag - all in sync and back to working as it has every day for months.

I haven't done anything in between (was out, away from pc) such as reboot router, sync PC clock.

I am well aware that anything is 'possible' and would have of course taken all steps.
But addressing each:
PC Clock - I have Windows Internet Time server, a widget that refreshes from different internet time clocks once per second, and another widget that does so every minute. And have done so reliably for months. So I think it reasonable to conclude all three didn't stop all at the same time and then all resolve their issues (also bear in mind I rebooted my PC, thus including these)
Router - I checked my ping, same as usual and all other network and internet fine. Again, the problem has resolved without reboot.

So I think it is a reasonable conclusion that none of the above decided to become problematic, was not corrected on PC reboot, and then miraculously repaired themselves.

What I do think is conspicuous is that this happened around the Open.
I start my platform at around 02:30 EST most days, As usual, it performed without issue until the Open. Then the problem began, cumulatively lagging.

A reboot of the PC did not fix.
Leaving the PC unattended and making no changes, and the problem has resolved itself.

This same thing has been observed before.
What we have here is not the platform i.e. indicators, 3rd party, level II etc etc because my platform has worked robustly since 21.1. and no changes.
The reason I have ChartLagTime loaded is because it is a visual sign which might otherwise not be noticed. I paid close attention and the chart/dom lag also an indication of something amiss was not the case - chart and dom in sync, but lagging real time. No sluggishness etc. As if doing market replay 5 minutes late and increasing. So the platform performance was not impaired.

I first turned to ChartLagTime when I experienced the occasional issue, as were others (for varying reasons). However mine were invariably one-offs, like this.

I was considering removing it since I have no changes to the platform I wish to make and have had no issues since 21.1, yet it is a useful visual for knowing the platform is in 'steady-state' and have caused no issues. I am confident today's incident was not platform chart lag etc of any sort as nothing has been changed from a setup that has run flawlessly for hours upon end. Neither do I have any reason to think it is internet, pc clock sync, router etc that has miraculously self-corrected

So are you suggesting this is a problem with the CQG feed?

I would appreciate your thoughts and suggestions....

Kind regards,

[NinjaTrader_PatrickG]

This points to a CQG feed issue other than the fact that it did not seem to universally affect all CQG-based connections which would have made a noticeable impact on our queue as most NinjaTrader users are using CQG-based connections.

[brucerobinson]

Indeed - I noted you had checked yours, and no other User reports.

Whilst the technicalities are above my pay grade, it does not sit well ‘specific to my data feed connection only’ either, insofar as I restarted both Ninja (hence the data feed connection) and PC/Ninja, and the problem persisted. Occasionally, for whatever reasons of networks, telecoms, and their software we get dropped connections, poor connections, lost packets whatever, but there was no indication of such, re-connections on restart of Ninja were normal I.e no delay in making the new connection on restart.

I’ve experienced it before, I’m confident the Open activity is not un-connected/co-incidental, yet there have been many many days much more extreme with zero issue. Doesn’t make sense.

Hence I do not know where to look, or what to conclude other than I cannot dispense with ChartLagTime to bring it to my attention.

Kind regards,

[NinjaTrader_Heath]

Hello Brucerobinson,

Thank you for your thorough investigation.

The issue you're happening likely isn't an issue of a full disconnect of you entire internet but much more likely is an issue with NinjaTrader connecting to the specific servers for market data required for your CQG connection.

As noted your other widgets and general internet did not disconnect but you only seem to be having issues with this one connection type which would imply a connection issue along your connection route to that specific server.

If you would like us to investigate further, please send me your log and trace files so that I may look into what occurred.

You can do this by going to the Control Center-> Help-> Email Support

Ensuring 'Log and Trace Files' is checked will include these files. This is checked by default.

Please reference the following ticket number in the body of the email: 2558188 ATTN Heath

Thank you I'll await those files.

[brucerobinson]

Hello Heath and thanks.
L&T sent as requested.

"which would imply a connection issue along your connection route to that specific server."
That would seem a plausible explanation.

I'm not an IT, software or programming specialist, but am reasonably 'savvy' so perhaps the following may eliminate possible causes.
Looking to identify the bottleneck of whatever is occasionally causing this:
Platform - in layman's terms - I've got lots of charts, data series etc etc etc but other than that no 'performance hitting' stuff in terms of data throughput such as tick replay etc. The long and short of it is that my setup has been performing without issue through very trying conditions. I haven't made any changes, don't have third party indicators, all the usual caveats and platform wasn't slow responding. Point being - not looking at whether my setup has anything to cause it, my setup has been working fine. It occurred on a single chart/single data series and 'exactly the same' as with my 'full workspace'. All that seems to point to 'slowness in getting the data, in order crunch it' rather than 'can't handle it fast enough'. And it appeared to make little difference whether I was using 1 data series/Instrument or 16......i.e. data to process was for 1 single chart only and still big cumulative delay.....
Connection to servers - I know what it is like with connection issues e.g. connecting at a weekend when Servers undergoing maintenance, or a poor/problematic internet connection etc. But it wasn't, on reboot of NT I looked for and saw nothing abnormal. Connection was 'snappy'.
Local Network - nothing to report. By the time I returned after having received Patrick's suggestions including reboot Router, Ninja delay had disappeared/corrected itself. Hence I haven't rebooted the router and my network has no issues and has not had. I will take a look at the router log in due course but 'nothing to see here'. I use a Microsoft app called Mouse Without Borders which allows one mouse to be used across multiple PCs & monitors using your LAN - point being it is the first thing to show up signs of any LAN issues with sluggish mouse performance - none at the time of the Ninja delay nor before nor since, & without Router reboot
Internet - everything else as normal , test loaded web pages, ran Speedcheck - down/upload & ping as normal. So I could load a webpage, ping Speedtest.net's servers no problem etc.

Yet Patrick and other Users didn't have problems with their CQG connections. Something happening in my 'tunnel' specifically to CQG servers interrupting data transfer and initiated by activity at the Open (I'm pretty sure on that latter, not co-incidental) but doesn't happen the rest of the last 3 months which have had days for more lively at the Open than yesterday.....

In the thread that gave birth to ChartTimeLag, at that time I had experienced an occasional incident just like this one (hence interest). I believe most of the User's issues reported their have been rectified due to 3rd party indicators, tick-intensive use such as tick-replay etc, plus coding improvements to the core platform's ability to process data in 21.1. I don't believe this is that sort of crunching or data throughput issue (single chart, single data series, remember.....)

Thanks for your input

Kind regards,

[NinjaTrader_PatrickG]

Your connection toggled between "normal" and "slow" quality for several hours on the 15th. Here are some examples from the beginning and end of when I detect this in your files:

• 2020-05-15 04:33:45:594 (AMP) CQG.Adapter.OnConnectionQualityChanged: serviceName='SecBuf' quality='Slow' host='cqginnyc035i.cqgnet.com'
• 2020-05-15 09:44:24:655 (AMP) CQG.Adapter.OnConnectionQualityChanged: serviceName='SecBuf' quality='Slow' host='cqginnyc035i.cqgnet.com'

This is likely the root cause of the symptom you experienced.

If you haven't done so already, I'd suggest contacting your broker about the potential of changing your account's 'preferred location' to one that is physically closer to you. This may mitigate intermittent networking issues related to the physical distance between you and CQG's servers.

Also, here are some general network troubleshooting steps we typically advise when low quality connection is the root cause of an issue:

• Restart NinjaTrader
• Restart your modem and (if applicable) router
• Restart your PC
• Clear your DNS
  • Open the Command Prompt
  • In the command prompt, run this command: ipconfig /flushdns
• If you are not already and the option is available to you, connect to the internet using an ethernet cable. Wireless connections are significantly more unreliable.
• Ensure exceptions have been added into any firewall/antivirus/backup programs for the Documents > "NinjaTrader 8" folder.
• Ensure that your computer is not turning off your Network Adapter to conserve power:
  • Right-click on your Start button
  • Select 'Device Manager'
  • Expand 'Network Adapters'
  • Right-click on your Network Adapter and select Properties > Power Management
  • Uncheck 'Allow the computer to turn off this device to save power' option
  • Click 'OK' to save the changes
  • Do this for all Network Adapters
• If the above steps do not help, contact your ISP and let them know you are experiencing a highly unstable connection. An issue of connectivity may exist on their end.

[brucerobinson]

Hello Patrick and thanks for investigating/pinning it down.
It all seems cohesive to my limited understanding - slow connection adequate for low data activity of the Overnight Session, overwhelmed by the Open causes cumulative lag consistent with your times and 'catches up' thereafter, or something akin.
AMP have servers in Cyprus, so that may be an option for 'preferred location'.
My speedtest.net pings would be to local nodes. I've pinged CQG's Chicago servers previously with a good connection and been good.

If you could indulge me further - Re: general network troubleshooting steps - reboot the whole lot, flush dns, I appreciate that and why this is best advice/practice, however the time required to do so would render the day useless as this issue seems to be prompted by the Open activity combined with occasional slow connection, and I trade primarily this time of day.
Would I be correct in thinking that simply disconnecting/reconnecting the internet connection from within my router (a 'soft' reconnect) may help (because it will 're-route' the various hops across the pond hopefully to a better quality one)?

Finally, is there a way of making what you find logged re: connection quality visible real time? The Connection 'traffic light' usually goes to Orange in case of poor connection quality, such as prior to an impending Connection Lost, but it is not doing so when I experience this issue (as I mentioned, it has happened previously occasionally, and I do not notice an Orange light). Hence Ninja Connection is reporting Connection Good, but under the hood says otherwise.

Simply put, you are reporting that under the hood it is identifiable as slow for 5 hrs, although the Connection light is reporting good connection Green, yet because of slow overnight activity it doesn't manifest in Platform performance until it gets tasty at the Open, then it all starts to back up. If I had some way of it being brought to my attention at 04:33:45:594 that would be helpful to a) not trade oblivious to the dodgy connection b) attempt rectify with a total reboot.

Kind regards,
Bruce

[NinjaTrader_PatrickG]

Each of the steps I provided is meant to be followed sequentially if symptoms do not improve. Simply disconnecting/reconnecting the internet connection from within your router may be all that is needed.

[brucerobinson]

Hello Patrick and thanks.
No answer to seeing what you find reporting connection ‘Slow’ real-time.
I recall another thread here relating to such an issue unearthed by ChartLagTime subsequently discussing software monitoring tools for network and perhaps internet latency, ill look there. Really need something with an alarm.
Perhaps other User may chime in.
Kind regards,

[NinjaTrader_PatrickG]

Thanks for the suggestion.

The internal tracking number for your feature request is SFT-4861. Please reference this internal tracking number when contacting Platform Support if you ever have questions regarding this feature request.

When a feature request is implemented, you'll find a description of the new feature in the release notes:

• http://ninjatrader.com/support/helpG...ease_notes.htm

[Johnny]

Did you find out what was causing this @brucerobinson?

I'm having some periodical issues with lagging charts. Nothing like you're describing, but I will literally see a quoted price on my right margin on chart trader and the charts lagging well behind.