Once NT6 data becomes delayed it doesn't catch up to be real-time

[dfogg]

With NT6 it has been observed many times that once data from eSignal becomes delayed, it does not catch up to be displayed in real-time. This is most easily seen by having a time and sales window up and comparing the time with a computer clock that shows seconds - the data to charts is also affected, it's just not so obvious. The data, both time and price, is correct, it just lags the real-time data from a few to many seconds.

This problem has been seen frequently when using an eSignal data feed with NT6 during fast market conditions. Falling behind is understandable, but failing to recover to show the data in real-time is not. Maybe the problem is specific to the NT6 to eSignal interface.

Disconnecting and reconnecting to eSignal results in the data being displayed in real-time. The result is similar to that in a previous post

http://www.ninjatrader-support.com/vb/showthread.php?t=820

which received the response:
"When you disconnect, you clear up an buffer on the MBT server side that may be building which could have caused delay. Then when you reconnect, all is back to normal. Any delay is on the data feed side not NT."

It appears there is a buffer in the interface that retains the data, it's not clear who wrote it or who is responsible for fixing it. Since winros was not shut down, it doesn't seem it's an eSignal problem.

The problem has not been noticed with NT5 connected to a Patsystems data feed, so NT5 is used for trading. Maybe NT6 wouldn't have the problem with the Patsystems data feed, but there is some doubt about that.

Today just for kicks, when swing highs or swing lows were noticed to have occurred with the eSignal "real-time" data, simulated orders were placed with NT6 - which was using it's "delayed" eSignal data. This made it pretty easy to make money. Would using data that's not caught up to real-time, make it more difficult to make money?

[NinjaTrader_Dierk]

There is no buffer in the eSignal interface. All data is routed to the UI as it comes in.

Note:
1) the issue only could be caused by NT if you see the NT process eating up all CPU cycles in the task manager (CPU is 100%, please make sure you then have not running any other apps)
2) please make sure you have a state-of-the art PC and at least 1 GB RAM. Low hardware resources could cause issue (1).
3) due to recent volume increase on markets data provider experienced "issues". Your problem likely could be related

[dfogg]

No buffer anywhere?

1. In response to:
"the issue only could be caused by NT if you see the NT process eating up all CPU cycles in the task manager (CPU is 100%, please make sure you then have not running any other apps)"

It is acknowledged that the CPU usage did reach 100%. So during that time, it is understandable that data can't be displayed in real-time.

However, later when the pulse of activity is over and the CPU usage is low, it is reasonable to expect NT6 to catchup and to display the data in real-time - The data displayed by other applications, such as eSignal, may be delayed for a while, but then they quickly go through the buffered data and start displaying the data in real-time.

No matter how fast the CPU is, there may be a case when the CPU usage reaches 100%. If this happens, the display of data needs to be handled gracefully and shown when possible in real-time.

2. In response to:
"There is no buffer in the eSignal interface. All data is routed to the UI as it comes in."

The current NT6 symptom is the same as if the data went into a FIFO buffer, since the charts and Time and Sales when they are updated by NT6, are just like charts and Time and Sales that were updated in real-time. It is difficult to envision how this accurate delay of data can occur without the data being buffered somewhere between the source of data, in this case eSignal, and the display of data by NT6.

[NinjaTrader_Dierk]

>> No matter how fast the CPU is, there may be a case when the CPU usage reaches 100%. If this happens, the display of data needs to be handled gracefully and shown when possible in real-time.
I agree. However as pointed out there is no buffering on NT's end. May be there is in eSignal client SW (which I don't know) and which would be out of our control anyway.

[dfogg]

Number of symbols appears to affect behavior

In Response to:
"However as pointed out there is no buffering on NT's end. May be there is in eSignal client SW (which I don't know) and which would be out of our control anyway."

Summary: NT6 with one or two symbols has been observed to fall behind and catch up. While with many, approximately 10, symbols it has been observed to fall behind and stay behind.

As a test NT6 and NT5 were both started on the same computer and both were connected to eSignal. In this case there were many symbols, charts and indicators displayed by NT6 while only one symbol with one chart and one T&S with NT5. In this case, the data for NT6 was noticed to stay delayed once it was delayed while that for NT5 would catch up to display the data in real-time.

As a second test the number of symbols on NT6 was reduced to two and though the data was observed to become delayed it would later catchup to display the data in real-time as desired. While adding symbols to NT5 resulted in it's data being delayed and not recovering.

Does that give you a hint as to the cause and means to the solve the problem?

[NinjaTrader_Dierk]

>> Does that give you a hint as to the cause and means to the solve the problem?
Unfortunately not, since NT5 and NT6 have exact same logic/code on handling eSignal data.

[BarDCR]

Regarding the problems you have been discussing, is this also what is making my Ninja Trader 6 so slow to load and populate charts? It is increasingly slower and is using in the hight 80% of my cpu. This was not a problem until a couple of weeks ago.
Additionally we have another system running Ninja Trader 6 and E-signal that when the market opens, the user has to disconnect then reconnect every morning.
Thanks

[NinjaTrader_Ray]

Are you guys running any custom/3rd party indicators on your charts?

[dfogg]

1. In response to Dierk:
">> Does that give you a hint as to the cause and means to the solve the problem?
Dierk: Unfortunately not, since NT5 and NT6 have exact same logic/code on handling eSignal data."

If the problem did not occur with NT5, I would agree and previously I hadn't noticed the problem with NT5. However, today it was observed that the problem could occur with NT5 when multiple symbols were present and not occur with NT6 with only two symbols present.

Since after the CPU usage has spiked to 100%, both NT5 and NT6 have sometimes demonstrated the problem when many symbols are present while both appear to function acceptably when only 1 or 2 symbols are present , isn't there a possibility that the source of the problem is with "NT5 and NT6 having the exact same logic/code for handling eSignal data"?

2. In response to Ray:
"Are you guys running any custom/3rd party indicators on your charts?"

On several of the NT6 charts there were custom indicators, as well as, a demo third party indicator. However, NT6, recovered as desired when the number of symbols was reduced to two with these indicators present.

3. As an observation, storing real-time bar data uses more CPU cycles, Memory, and I/O than expected. Even tonight when the market is not very active and with only two symbols up, NT6 consistently reads and writes 1 to 2 megabytes with an interval of 60 seconds between the transfers. When the transfer is made, the CPU usage for NT6 spikes from less than 3 percent to the area of 70-80% and the memory used increases by approximately 3 megabytes.

This large data transfer, memory increase, and CPU usage was greatly diminished by turning off
Store real-time bar data
on the
Tools/Options/Data page.

Surprisingly whether or not
Run Market Replay Recorder
is on or off wasn't observed to have any significant effect.

The above effects are easily seen by running:
Process Explorer for Windows v10.21
available for free from:

http://www.microsoft.com/technet/sysinternals/Utilities/ProcessExplorer.mspx

Within the Process Explorer window, Double click on NinjaTrader.exe and a separate window will open showing the above information.

[NinjaTrader_Dierk]

- please review performance optimization considerations here: http://www.ninjatrader-support.com/H...anceTips1.html
- in particular: since you are on eSignal, there is no need to have "Store real-time bar data" checked (this will overcome issue (3))

[NinjaTrader_Ray]

What 3rd party demo indicator are you running? Also check the custom indicators by removing them to see if that makes a difference.

[BarDCR]

Thanks Ray,

We are running Advanced Trading Workshop indicators and Woodies CCI.
Thanks