Kinetick on NinjaTrader is slow compared to ThinkOrSwim

[molecool]

One of the reasons why I don't use NinjaTrader for much of my charting is because it's SLOW.

Compare the two enclosed video clips I uploaded to my site:

Ninja Clip

TOS Clip

Every morning before I put up a post on Evil Speculator I scan through dozens of charts on a hunt for valid setups. I keep trying to use Ninja because *I prefer using it* but it's simply too slow. I can't get into the flow because the data feed is lagging. Now compare it with TOS which is blazingly fast. Same connection, same system. What gives?

I complained about this about a year ago and frankly there is nothing else dragging down Ninja on my system - it's clearly the data connection. My trading buddy in Australia is complaining about the same thing. Is it something in the settings? Clearly Kinetick should be FASTER than a FREE service provided by TOS, right? I tried both the Kinetick and the IQFeed connectors. I tried to close all other open charts. Nothing doing. I also see the same lag on a SUPER fast pimped out dedicated production system we are running in a server farm in Florida - super fast pipe - nothing else running - 32GB of RAM. It's as fast as they come and I think that proves my point.

If this could be addressed I would be able to dump TOS and only use Ninja for charting. That also would mean that you would see more NinjaTrader charts on Evil Speculator ;-)

Any help/pointers/suggestions that truly address this problem would be appreciated.
Michael

[NinjaTrader_Brandon]

Hello molecool,

Thanks for your post.

The only suggestions I could make would be to review the Performance Tips from the help guide: http://www.ninjatrader.com/support/h...ance_tips2.htm

Additionally please note that performance will be enhanced with NinjaTrader 8.

[molecool]

Hello molecool,

Thanks for your post.

The only suggestions I could make would be to review the Performance Tips from the help guide: http://www.ninjatrader.com/support/h...ance_tips2.htm

Additionally please note that performance will be enhanced with NinjaTrader 8.

Okay, let's see:

Variables that can impact market data latency
• Connecting wirelessly can drastically impact performance during heavy volume periods. Depending on signal strength, your modem/router is unable to push all the data through. This can be resolved by connecting directly into the modem/router via a hard line.

Connected via RJ45 to a high speed network at a colo _ I get 94Mbit/sec up/down. Here's the speed test:

Speed Test: http://screencast.com/t/hnnjtisxMP

I don't think one could ask for more optimal conditions.



• Serious abnormal increases in market volume

Nope - just tested before the open. Very little market activity.

• Data provider servers could be lagging

Data provider is Kinetick - connected via IQFeed adaptor. Same happens via Kinetick adaptor.

• Limited bandwidth internet service (56K dial up modem is not acceptable for example)

Uuuh - no ;-)
• Inadequate PC hardware or running too many applications on your PC

I have an inkling my server is fast enough:

Server Stats: http://screencast.com/t/4Gz2huLA



I would really like to sort this out with you guys. Is this a Kinetick or a NinjaTrader issue? You guys are testing NinjaTrader8 right now. Fine, can you perhaps send me a video where you're pulling a few charts just like the ones I made?

[NinjaTrader_Matthew]

Hi Mole,

I've put together two videos, the first demonstrates the response time I receive using data which has been previously loaded and cached:

http://screencast.com/t/1yhxR8La0LI

The second video here is with a clean db folder, with a number of stocks pulled at random:

http://screencast.com/t/IuN8WOPTBGR

I'm not sure what the difference between our setups would be in our testing. To be clear, I am requesting 30 days of minute data. If you are asking for much more, you may want to reduce the look back on the charts you plan on switching frequently to reduce the load times.

In my tests the download of data does not take much time, even with up to 365 days of data I feel the response time is reasonable. Please also keep in mind that if you have already downloaded historical data for an instrument, based on our data loading rules, NinjaTrader will only request data for a period that is missing and not the entire lookback if it's already in your database, which improves the chart load time when switching instruments. Details on these load rules can be found below under 'When does NinjaTrader load historical data?'

http://www.ninjatrader.com/support/helpGuides/nt7/historical_data.htm

However, what you very well could be seeing is not the load time of the historical data/Kinetick, but with the processing of the indicators. I do not know how TOS works under the hood, but NinjaTrader will process all of your indicators starting on the first bar requested. Depending on the number of days to load, it could slow down the response time of the chart as it calculates your indicators on each data point. As we all know NinjaTrader UI is limited to a single thread and I suspect TOS is multi-threaded, but I don't have that information in front of me.

I'd also suggest running your tests again without any indicators to isolate the slow load time. Although as you can see in the videos, I've used a few system indicators and it does not add much of a performance impact on the load time. You may want to run your tests with my setup and compare versus any 3rd party indicators to see if you can identify a bottleneck.

Another tip would be using the Market Analyzer to cache your historical data, which would for sure improve chart performance. As long as you have added an indicator column to your Market Analyzer, it will download the requested # of bars configured in the Market Analyzer and will store that in the cache. This data will be downloaded at startup and will continue to cache the data as it comes in real-time, which will also improve the amount of time it takes to switch charts. (Note,there is a hard coded limit of 100 bars, so any values over this number will be ignored for caching purposes).

I hope that helps and let me know if you have any questions.

[molecool]

Awesome response - is this NT7? Quite a difference! Let me comment inline below:

Hi Mole,

I've put together two videos, the first demonstrates the response time I receive using data which has been previously loaded and cached:

http://screencast.com/t/1yhxR8La0LI

The second video here is with a clean db folder, with a number of stocks pulled at random:

http://screencast.com/t/IuN8WOPTBGR

Wow, that's a hell of a lot faster than what we (my trading buds and I) are witnessing. Clearly it's a lot faster than what I'm seeing on my end. I just tried to pull 300 60-minute bars of NG ##-## on the production box and it took 30 seconds!

I'm not sure what the difference between our setups would be in our testing. To be clear, I am requesting 30 days of minute data. If you are asking for much more, you may want to reduce the look back on the charts you plan on switching frequently to reduce the load times.

Indeed! I'm all about identical test conditions. I was pulling 300 bars of 60-minute data. I think that's shown in my video. Same on the TOS side - actually TOS pulls daily data as well at the same time (and is still faster - cough cough).

In my tests the download of data does not take much time, even with up to 365 days of data I feel the response time is reasonable. Please also keep in mind that if you have already downloaded historical data for an instrument, based on our data loading rules, NinjaTrader will only request data for a period that is missing and not the entire lookback if it's already in your database, which improves the chart load time when switching instruments. Details on these load rules can be found below under 'When does NinjaTrader load historical data?'

http://www.ninjatrader.com/support/helpGuides/nt7/historical_data.htm

Yes, that's pretty much what I thought. Look, I go through the same symbol list every single morning. So that data has been loaded over and over again. It's slow though as you can see from my video.

However, what you very well could be seeing is not the load time of the historical data/Kinetick, but with the processing of the indicators. I do not know how TOS works under the hood, but NinjaTrader will process all of your indicators starting on the first bar requested. Depending on the number of days to load, it could slow down the response time of the chart as it calculates your indicators on each data point. As we all know NinjaTrader UI is limited to a single thread and I suspect TOS is multi-threaded, but I don't have that information in front of me.

It's complicated on the Java side - TOS has been written in Java. It's a massive resource hog which is why I hate using TOS. But I do think it's multi-threaded, otherwise they wouldn't be able to do things so fast. The indicators I use in that workspace are pretty lightweight - I have bollingers running and one I coded on my own called Net-Lines. I can send you the source - it's very small. Here are the rules:

http://evilspeculator.com/tools/cheat-sheet-2/

Very little processing involved in doing that as you can imagine. Otherwise I don't know what could drag it down. BTW, I also have written the Net-Lines indicator in TOS - clumsy crap code due to ThinkScript sucking like [ahem] and it's still fast.

I'd also suggest running your tests again without any indicators to isolate the slow load time. Although as you can see in the videos, I've used a few system indicators and it does not add much of a performance impact on the load time. You may want to run your tests with my setup and compare versus any 3rd party indicators to see if you can identify a bottleneck.

You are right - much faster without indicators. I be damned:

Ninja without indicators.

Now it did hang there on gold and silver for a little. But to me it seems quite a bit faster.

TOS doesn't suffer from an indicator delay fortunately. So NT8 will fix that as it's multi-threaded?

Another tip would be using the Market Analyzer to cache your historical data, which would for sure improve chart performance. As long as you have added an indicator column to your Market Analyzer, it will download the requested # of bars configured in the Market Analyzer and will store that in the cache. This data will be downloaded at startup and will continue to cache the data as it comes in real-time, which will also improve the amount of time it takes to switch charts. (Note,there is a hard coded limit of 100 bars, so any values over this number will be ignored for caching purposes).

I hope that helps and let me know if you have any questions.

I am actually using the market analyzer - that's how I step through my symbol list. But I have not tried to add indicators as separate columns in that workspace. I can try that to see if it speeds it up. Any indicator will do or do I need to use the ones on my chart?

Excellent input - many thanks.

[NinjaTrader_Matthew]

Yes, that was NinjaTrader 7.

Any indicator will do, that is all that is needed for the Market Analyzer to start pulling and caching historical data - just something like an SMA which uses historical data will work. That's good to hear you already use the Market Analyzer and have a dedicated symbol list, this tells me the response time has very little to do with Kinetick and is more of UI/Indicator issue.

The NinjaTrader 8 UI runs on a separate thread, so your NS objects will be able to process separately which will help. Whether or not your NT8 NS indicators are multi threaded or not will ultimately depend on how they're programmed. Our internal/system methods will be.

Some food for thought: While we look forward to the overall performance improvements in NinjaTrader 8, this does not necessarily mean that all NS Indicators will absolutely take advantage of these improvements. If the 3rd party developers were not mindful of multi-threaded considerations when the indicators is particularly resource intensive to begin with, users could theoretically be stuck with an inefficient indicator. This is something we plan on documenting 'best practices' to ensure users are not stuck with legacy implementation when they can be improved - but we cannot gurantee that.

[molecool]

Yes, that was NinjaTrader 7.

Any indicator will do, that is all that is needed for the Market Analyzer to start pulling and caching historical data - just something like an SMA which uses historical data will work. That's good to hear you already use the Market Analyzer and have a dedicated symbol list, this tells me the response time has very little to do with Kinetick and is more of UI/Indicator issue.

The NinjaTrader 8 UI runs on a separate thread, so your NS objects will be able to process separately which will help. Whether or not your NT8 NS indicators are multi threaded or not will ultimately depend on how they're programmed. Our internal/system methods will be.

Some food for thought: While we look forward to the overall performance improvements in NinjaTrader 8, this does not necessarily mean that all NS Indicators will absolutely take advantage of these improvements. If the 3rd party developers were not mindful of multi-threaded considerations when the indicators is particularly resource intensive to begin with, users could theoretically be stuck with an inefficient indicator. This is something we plan on documenting 'best practices' to ensure users are not stuck with legacy implementation when they can be improved - but we cannot gurantee that.

I've added a LogReturn (very lightweight) indicator to my list but only see a modest improvement. It's better than nothing I guess. Here's a clip that shows me walking up/down my list. As you can see it's snappier if I only load hourly data - but sometimes it still gets stuck although I just loaded the same symbol a few seconds earlier. If I add the daily panel as well then things really slow down. HOWEVER the daily panels on average are a lot faster than the minutes panels - can you confirm that?

NT8 - I've implemented separate threading into several of my strategies (e.g. notification and logging threads) but not for my indicators. A best practices document would help a lot. As a matter of fact - we could do a web seminar showing the difference between a threaded and non-threaded indicator. If we ever get NT8 that is ;-)

[NinjaTrader_Matthew]

Yes, I'd expect daily bars to load faster than minute bars. There is less data to process.

edit: To clarify, if we're comparing 'days to load', daily bars would be load faster. If you're requesting the same number of bars, it should be roughly the same.

[NinjaTrader_Brandon]

Hello molecool,

Thanks for your reply.

As far as the loading of minute data vs daily data it really just comes down to how many bars are being loaded. If they are loading the same number of bars I would expect them to take about the same amount of time.

[koganam]

I've added a LogReturn (very lightweight) indicator to my list but only see a modest improvement. It's better than nothing I guess. Here's a clip that shows me walking up/down my list. As you can see it's snappier if I only load hourly data - but sometimes it still gets stuck although I just loaded the same symbol a few seconds earlier. If I add the daily panel as well then things really slow down. HOWEVER the daily panels on average are a lot faster than the minutes panels - can you confirm that?

NT8 - I've implemented separate threading into several of my strategies (e.g. notification and logging threads) but not for my indicators. A best practices document would help a lot. As a matter of fact - we could do a web seminar showing the difference between a threaded and non-threaded indicator. If we ever get NT8 that is ;-)

Not quite a valid comparison as far as data feeds are concerned. TOS is snapshot data: Kinteck is raw data. Very different beast as far as load is concerned. Look at the time and sales data, and you can clearly see the source of the difference in data load.

cf: My response, http://www.ninjatrader.com/support/f...d.php?p=264817 in this thread, http://www.ninjatrader.com/support/f...d.php?p=246893, which discusses essentially the same issue in 2011.