Responsiveness of indicators

[pjsmith]

Hi,


I have a general query. I use a third party DOM with NT8. When the tick count is very hi, that DOM lags the action by perhaps 3-5 seconds. I'm pretty sure I have an indicator (my own) that might also be effected.

BUT, the NT DOM running side by side, shows the correct pricing in real-time. I have verified this with another data feed and by executing live limit orders against it.

So, my question is really a design one, which I would not be able to answer myself and I need your input on.

Is OnMarketData fed to the DOM internally in a different and perhaps higher performance way than to an indicator? Would you expect the DOM to be more responsive than perhaps the marketanalyzer indicators?

Thanks for any input.

P.S. On an unrelated matter - Well done with b12 - I know from using it this week it seems to perform much better than previous builds when switching between chart tabs, scrolling charts, etc. Still early days, but I'd say stability and performance has certainly improved.

[NinjaTrader_ChrisL]

Hello.

Thank you for the post. We're glad to hear you are enjoying NT8 R12.

Since the core of NinjaTrader is not open source, it can not be determined what kind of logic the SuperDOM uses.

Likely, it is similar to this sample, where OnMarketData is used to update a custom price ladder:

https://ninjatrader.com/support/forum/showthread.php?t=3478

If the third party DOM is being slow, there is likely additional logic working that the default SuperDOM does not have. Any errors that occur while using this DOM must be addressed by the developer of the custom DOM.

Please let us know if you have any questions.

[pjsmith]

Hi,

Thanks for your answer. As the developers of the product though (assuming you do develop it and it's not developed for you by third parties), I thought you might be able to answer this query. Is it likely that the NT intergrated DOM can achieve performance a coded indicator cannot via onMarketdata? I really would appreciate you asking your developers if this is the case. Thank you for the example. It is in fact what I based some of my own code that experiences the lag on. Also, the other DOM is a commercial product and it experiences same.

Please note, I do not want to see the source, nor ask how it works, just the answer to the above query. I understand it is closed source. The reason for this is simply that if your chaps say 'no', you should be able to achieve similar performance via your own code, then I can work to resolve any issues. If there are internal reasons why the integrated DOM seems to be able to outperform third party indicators, then I a fighting a lost cause, so can avoid wasting my time.

If you could pose this query to the developers, I would appreciate it.

[NinjaTrader_Brett]

NinjaTrader 8 has multiple instrument threads and multiple UI threads (Matching the number of cores in your computer). The SuperDOM and any NinjaScript are on equal footing when it comes to processing market data.

What can happen is that a UI / Instrument thread must finish processing a tick before it can move on to the next. If a particular UI or Instrument thread has a lot of processing to do and does not complete before the next tick comes in then any UI's being driven by this thread will be bogged down and start to fall behind.

When each window is opened its randomly assigned one of the UI threads in the thread pool of which to process on. So just by chance the NinjaTrader SuperDOM likely is on a different thread then the UI thread your NinjaScript window is running on. Hence why the NinjaTrader DOM appears to keep up.

Your symptoms are an indication that a UI thread is getting bogged down and optimizations need to be applied to code running on that UI thread. We spent a lot of time on optimization with the NinjaTrader 8 SuperDOM.

[pjsmith]

Thanks for the explanation. That helps my understanding of the issue a lot.

[pjsmith]

OK, I tried reducing the number of indicators etc., but i need some real help finding out where the lags are in this setup. Today in all this action, the DOM and charts were 30 points (yes, points!) behind. My separate feed showed the correct pricing.

How do I profile this to tell where/what is causing the lags? The data feed? A third party tool? Please don't ask me to experiment by simply removing them all one by one. My setup is as I need it and I just need to identify what is generating the lag as days like today a pretty much untraceable, at least, unless you do it blind!

See attached pic. And my other feed showed yet a different (and correct) price. I have the setup how I want it. I've shut down everything I can. My PC is an 8 core machine running W10. Becuase some of the windows, charts, DOM's are all showing different prices and different lags, It's clear it's due to my setup/nt I think, and not the PC. At times, the NT superdom is _almost_ correct, but like today, when it's really fast, NT is just a mile behind the action. Running on Kinnetick data feed btw.

Any assistance as to anything to try to profile what in the NT setup is causing the lag would be appreciated. Unless I can nail this, I have to bail on this platform as I simply cannot trade with it. Please note - I have it how I want it and it works. I _want_ to keep it!

If we can come to a conclusion about the reason for the delay, I can get a new PC, new data feed, approach vendors with evidence, etc. Just need some help to get that evidence and nail down the cause.

Thanks.

[edit - I've uploaded another example from just now, showing NT superdom, third party dom, and actual pricing from IB TWS (which is correct)]

[newuserabc]

OK, I tried reducing the number of indicators etc., but i need some real help finding out where the lags are in this setup. Today in all this action, the DOM and charts were 30 points (yes, points!) behind. My separate feed showed the correct pricing.

How do I profile this to tell where/what is causing the lags? The data feed? A third party tool? Please don't ask me to experiment by simply removing them all one by one. My setup is as I need it and I just need to identify what is generating the lag as days like today a pretty much untraceable, at least, unless you do it blind!

See attached pic. And my other feed showed yet a different (and correct) price. I have the setup how I want it. I've shut down everything I can. My PC is an 8 core machine running W10. Becuase some of the windows, charts, DOM's are all showing different prices and different lags, It's clear it's due to my setup/nt I think, and not the PC. At times, the NT superdom is _almost_ correct, but like today, when it's really fast, NT is just a mile behind the action. Running on Kinnetick data feed btw.


Any assistance as to anything to try to profile what in the NT setup is causing the lag would be appreciated. Unless I can nail this, I have to bail on this platform as I simply cannot trade with it. Please note - I have it how I want it and it works. I _want_ to keep it!

Thanks.

contact the vendor

[pjsmith]

contact the vendor

The lag is in NT, as a whole, not just the third party dom. And, I have contact them too. I have replicated this issue as I stated in my own code. The issue also exists in the NT superDOM and charts.

[NinjaTrader_ChrisL]

Hello pjsmith.

Thank you for the follow-up.

I would like to contact the vendor myself for this item to gain insight into the way this DOM is implemented and possibly give them advice on what might need to change about this DOM to make it more efficient.

Could you please reply to this thread with the name of the vendor and the name of this DOM product?

Also, so that I can address this item on a more focused and personal level, please write into platformsupport[at]ninjatrader[dot]com with your email and reference the following in the body of the email: "Attn ChrisL <Paste the link to this forum thread here>".

Thanks in advance. I look forward to resolving this item.

[pjsmith]

Thanks. I'll send those details now. As I am not sure the DOM is actually to blame, I'd rather not name them here, but I will send full details in the email along with the ticket references I have open with them too, in case it helps. Thanks again.

[sledge]

The lag is in NT, as a whole, not just the third party dom. And, I have contact them too. I have replicated this issue as I stated in my own code. The issue also exists in the NT superDOM and charts.

Get a faster connection?

Are you overseas perhaps? There are more than a few posts about lag time during fast moving markets like today.

[pjsmith]

Hi Sledge,

Thanks for the suggestion. In part, you would be right, but that would only account for 70ms or so of lag. Not 10 minutes My charts actually continued playing data back almost 10 minutes after the hard close last night. (the nt superdom finished much sooner)

I have a theory, just not sure if it's valid and awaiting contact from NT support directly. I'm guessing, but having skimmed a bit of documentation, most indicators, windows, ui, code, etc that refers to a single instrument (in this case ES) are on the same thread perhaps. Links in a chain so to speak. Each tick must pass through the chain until the next is processed. When it gets really busy and if you have a few links to go through (multiple time frame charts, indicators etc), ticks cannot be processed as fast as they are coming in and they get queued, essentially for playback as fast as possible. The NT superdom seems to run I think on it's own thread.

The data delay idea doesn't stand up as on Friday ( and in part yesterday), the NT superdom was showing all the correct live data with little lag, whilst only the other charts and indicators were behind.

I appreciate these are extreme examples, but I've had this again and again. It may well be that my kit or what I am trying to do with NT with multiple charts and expecting real-time performance is simply not possible, which is OK too (if not disappointing). Just need to know.

[pjsmith]

OK, so I am actually seeing the same issue on and off this morning, which I did not expect. ES is certainly jumping around a lot, but still... Anyhow. I have one es chart that has tick replay on. Just one. I just turned tick replay off and ALL es charts now appear more responsive. I'm not sure if this is a fix, but I have not seen the charts update this fast in a few days. CPU usage appears to have dropped too. I will test further, but could it be that tick replay enabled in one window affects all other windows/code for that instrument? That would not make sense to me, but perhaps the NT techies know better?